Defense Business Systems: DOD Needs to Continue Improving Guidance and Plans for Effectively Managing Investments
Released: 2018-04-16T00:00:00

Letter

Background

DOD is the largest U.S. federal department and one of the most complex organizations in the world. In support of its military operations, the department manages many interdependent business functions, including logistics management, procurement, health care management, and financial management. DOD relies extensively on IT to support its business functions.

According to its IT investment data, the department has 2,097 business system investments. The department’s fiscal year 2018 IT budget request states that DOD plans to spend about $8.7 billion in fiscal year 2018 on its defense business systems.

The IT budget organizes investments by mission areas. The four mission areas are enterprise information environment, business, warfighting, and defense intelligence. Figure 1 shows the amount of DOD’s requested fiscal year 2018 IT budget that the department plans to spend on each mission area.

The department further organizes its IT budget by segments. For example, the business mission area includes segments such as financial management, health, and human resource management. Figure 2 shows the department’s projected fiscal year 2018 spending for each segment in the business mission area.

GAO designated the department’s business systems modernization efforts as high risk in1995 and has continued to do so in the years since. DOD currently bears responsibility, in whole or in part, for half of the programs (17 of 34 programs) across the federal government that we have designated as high risk. Seven of these areas are specific to the department, and 10 other high-risk areas are shared with other federal agencies.

Collectively, these high-risk areas are linked to the department’s ability to perform its overall mission and affect the readiness and capabilities of U.S. military forces. DOD’s business systems modernization is one of the department’s specific high-risk areas and is essential for addressing many of the department’s other high-risk areas. For example, modernized business systems are integral to the department’s efforts to address its financial and supply chain high-risk areas.

Since 2005, we have issued 11 reports in response to mandates directing GAO to assess DOD’s actions to respond to business system modernization provisions contained in Section 2222 of Title 10, United States Code. These reports contained 23 recommendations to help strengthen the department’s management of its business systems. As of September 2017, the department had implemented 13 of the recommendations and 2 had been closed as not implemented. The other 8 recommendations remain open. The 11 reports are listed in appendix II.

The NDAA for Fiscal Year 2016 Included Provisions for Managing Defense Business Systems

The NDAA for Fiscal Year 2016 included provisions requiring DOD to perform certain activities aimed at ensuring that its business system investments are managed efficiently and effectively. Specifically, the act established requirements for the department related to issuing policy and guidance for managing defense business systems; developing and maintaining a defense business enterprise architecture; establishing a Defense Business Council to provide advice to the Secretary on managing defense business systems; and obtaining approvals before systems proceed into development (or if no development is required, into production or fielding) and related annual reviews.

According to the Joint Explanatory Statement accompanying the NDAA for Fiscal Year 2016, the act revised Section 2222 of Title 10, United States Code, to streamline requirements and clarify the responsibilities of senior officials related to acquiring and managing business systems. Key revisions pertain to:

Covered defense business systems. The code previously defined a covered defense business system as a system having a total cost of over $1 million over the period of the future-years defense program.

As revised, the code now defines a covered defense business system as a system that is expected to have a total amount of budget authority over the period of the current future-years defense program of over $50 million.

Priority defense business systems. The act established a new category of system, called a priority defense business system. This is a system that is (1) expected to have a total amount of budget authority of over $250 million over the period of the current future- years defense program, or (2) designated by the DCMO as a priority defense business system based on specific program analyses of factors including complexity, scope, and technical risk, and after notification to Congress of such designation.

Thresholds and officials responsible for review and certification of defense business systems. The code previously stated that the DCMO had responsibility for reviewing and certifying all defense business system investments over $1 million over the future-years defense program. The revised code states that, unless otherwise assigned by the Secretary of Defense, military department Chief Management Officers (CMO) are to have approval authority for their covered defense business system investments below $250 million over the future-years defense program. The DCMO is to have approval authority for defense business systems owned by DOD components other than the military departments, systems that will support the business process of more than one military department or other component, and priority defense business systems.

Certification requirements. The code previously required that a defense business system program be reviewed and certified, at least annually, on the basis of its compliance with the business enterprise architecture and appropriate business process reengineering. In addition to these requirements, the revised code requires that the business system program be reviewed and certified on the basis of having valid, achievable requirements and a viable plan for implementing the requirements; having an acquisition strategy designed to eliminate or reduce the need to tailor commercial off-the- shelf systems; and being in compliance with the department’s auditability requirements.

Key Roles and Responsibilities for Managing Defense Business Systems

DOD Instruction 5000.75: Business Systems Requirements and Acquisition assigns roles and responsibilities for managing defense business system investments. Table 1 identifies the key entities and their responsibilities for managing defense business system investments.

DOD Has Made Progress in Complying with Legislative Provisions for Managing Defense Business Systems, but More Remains to Be Done

DOD has taken steps to address provisions of the NDAA for Fiscal Year 2016 related to defense business system investments. Specifically, as called for in the act, the department has established guidance that addresses most legislative requirements for managing its defense business systems; however, the military departments are still developing guidance to fully address certification requirements for their systems.

Further, DOD has developed a business enterprise architecture and is in the process of updating the architecture to improve its content. The department also has a plan to improve the usefulness of the business enterprise architecture; however, the department has not delivered the plan’s intended capabilities. In addition, the department is in the process of updating its IT enterprise architecture; however, it does not have a plan for improving the department’s IT and computing infrastructure for each of the major business processes. Further, the department has not yet demonstrated that the business enterprise architecture and the IT enterprise architecture are integrated.

The department fully addressed the act’s requirement related to defense business system oversight. Specifically, the department’s governance board, called the Defense Business Council, addressed legislative provisions to provide advice to the Secretary of Defense.

Lastly, DOD and the military departments did not apply new legislative requirements when certifying business systems for fiscal year 2017. Instead, the DOD DCMO certified the systems in our sample in accordance with the previous fiscal year’s (fiscal year 2016) certification requirements.

DOD Issued Guidance Addressing Most Legislative Requirements for Managing Business Systems

The NDAA for Fiscal Year 2016 required the Secretary of Defense to issue guidance by December 31, 2016 to provide for the coordination of, and decision making for, the planning, programming, and control of investments in covered defense business systems. The act required this guidance to address six elements:

Policy to ensure DOD business processes are continuously reviewed and revised to implement the most streamlined and efficient business processes practicable and eliminate or reduce the need to tailor commercial off-the-shelf systems to meet or incorporate requirements or interfaces that are unique to the department.

A process to establish requirements for covered defense business systems.

Mechanisms for planning and controlling investments in covered defense business systems, including a process for the collection and review of programming and budgeting information for covered defense business systems.

Policy requiring the periodic review of covered defense business systems that have been fully deployed, by portfolio, to ensure that investments in such portfolios are appropriate.

Policy to ensure full consideration of sustainability and technological refreshment requirements, and the appropriate use of open architectures.

Policy to ensure that best acquisition and systems engineering practices are used in the procurement and deployment of commercial systems, modified commercial systems, and defense-unique systems to meet DOD missions.

Of these six elements called for by the act, the department has issued guidance that fully addresses four elements and partially addresses two elements. Table 2 summarizes our assessment of DOD’s guidance relative to the act’s requirements.

DOD fully addressed the element requiring policy to ensure that the business processes of the department are continuously reviewed and revised. For example, DOD Instruction 5000.75 requires the functional sponsor of a defense business system to engage in continuous process improvement throughout all phases of the business capability acquisition cycle. The department also fully addressed the element to provide a process for establishing requirements for covered defense business systems with DOD Instruction 5000.75, which introduces the business capability acquisition cycle for business system requirements and acquisition.

In addition, DOD fully addressed the element to provide mechanisms for planning and controlling investments in covered defense business systems. Specifically, the department’s Financial Management Regulation; Directive 7045.14 on its planning, programming, budgeting, and execution process; and the April 2017 Defense Business System Investment Management Guidance provide such mechanisms. For example, the April 2017 investment management guidance includes a process, called the integrated business framework, which the department is to follow for selecting, managing, and evaluating the results of investments in defense business systems. In addition, the directive assigns the DOD CIO responsibility for participating in the department’s annual resource allocation process and for advising the Secretary and Deputy Secretary of the Defense on IT resource allocations and investment decisions.

Further, DOD fully addressed the requirement for a policy requiring the periodic review of covered business systems that have been fully deployed, by portfolio, to ensure that investments in such portfolios are appropriate. Specifically, the department’s April 2017 Defense Business System Investment Management Guidance requires the department to annually review an organization’s plan for managing its portfolio of defense business systems over the period of the current future-years defense program (e.g., Army’s plan for its financial management systems) to ensure, among other things, that the portfolio is aligned with applicable functional strategies (e.g., DOD’s strategy for its financial management functional area).

DOD partially addressed the element requiring policy to ensure full consideration of sustainability and technological refreshment requirements, and the appropriate use of open architectures. Specifically, the department established policy requiring consideration of open architectures, but it has not established policy requiring consideration of sustainability and technological refreshment requirements. The Office of the DCMO stated that future guidance is expected to provide a policy to ensure full consideration of sustainability and technological refreshment requirements. However, the department could not provide a time frame for when the guidance will be developed and issued. Without a policy requiring full consideration of sustainability and technological refreshment requirements for its defense business system investments, the department may not be able to ensure that it has a full understanding of the costs associated with these requirements. As a result, the department may not be able to effectively manage spending on these systems.

DOD has also partially addressed the element requiring policy to ensure that best acquisition and systems engineering practices are used in the procurement and deployment of commercial, modified-commercial, and defense-unique systems. Specifically, the department has established policy requiring the acquisition of business systems to be aligned with commercial best practices and to minimize the need for customization of commercial products to the maximum extent possible. On the other hand, the department has not established policy to ensure the use of best systems engineering practices.

With regard to this finding, officials in the Office of the DCMO asserted that DOD Instruction 5000.75 addresses the requirement. However, while the instruction requires the system acquisition strategy to include a description of how the program plans to leverage systems engineering, it does not require the use of best systems engineering practices. Without a policy requiring the use of best systems engineering practices in the procurement and deployment of commercial, modified, and defense- unique systems, the department may be limited in its ability to effectively balance meeting system cost and performance objectives.

DOD Issued Guidance that Addresses New Certification Requirements, and the Military Departments Have Made Mixed Progress in Issuing Supporting Guidance

In addition to guidance for addressing the aforementioned legislative requirements for business systems management, the NDAA for Fiscal Year 2016 requires the Secretary to direct the DCMO and the CMO of each of the military departments to issue and maintain supporting guidance, as appropriate and within their respective areas of responsibility. In this regard, one of the key areas for which the DCMO and military department CMOs are to provide supporting guidance is the review and certification of defense business systems in accordance with specific requirements.

Specifically, the act requires that, for any fiscal year in which funds are expended for development or sustainment pursuant to a covered defense business system program, the appropriate approval official is to review the system to determine if the system: has been, or is being, reengineered to be as streamlined and efficient as practicable, and whether the implementation of the system will maximize the elimination of unique software requirements and unique interfaces; is in compliance with the business enterprise architecture or will be in compliance as a result of planned modifications; has valid, achievable requirements, and a viable plan for implementing those requirements (including, as appropriate, market research, business process reengineering, and prototyping activities); has an acquisition strategy designed to eliminate or reduce the need to tailor commercial off-the-shelf systems to meet unique requirements, incorporate unique requirements, or incorporate unique interfaces to the maximum extent practicable; and is in compliance with the department’s auditability requirements.

The act and DOD Instruction 5000.75 define the systems that the DOD DCMO is responsible for certifying and the systems that military department CMOs are responsible for certifying.

Consistent with the act, in April 2017, the DCMO issued guidance for certifying officials that addresses the certification requirements. Table 3 provides our rating and assessment of the DCMO’s guidance for implementing defense business system certification requirements.

By establishing guidance requiring that defense business systems be certified on the basis of the legislative requirements, the department is better positioned to ensure that a covered system does not proceed into development (or, if no development is required, into production or fielding) without the appropriate due diligence. Further, the department has taken steps which should help ensure that funds are limited to systems in development or sustainment that meet these requirements.

Air Force and Navy Guidance Partially Addresses Certification Requirements; Army Has Not Yet Issued Guidance

The military departments have made mixed progress in developing supporting guidance to assist in making certification decisions regarding systems within their respective areas of responsibility. More specifically, the Air Force has issued supporting guidance that addresses three of the act’s five certification requirements, but does not address the remaining two requirements. Navy has issued guidance that addresses two of the certification requirements, partially addresses one requirement, and does not address two requirements. The Army has not yet issued guidance on any of the five certification requirements.

Table 4 provides an overview of our assessment of the Air Force’s, Navy’s, and Army’s guidance relative to the NDAA for Fiscal Year 2016 certification requirements. Each department’s efforts are further discussed following the table.

Air Force. In April 2017, the Department of the Air Force issued guidance for certifying business systems for fiscal year 2018. The guidance addresses the requirements that a system be certified on the basis of sufficient business process reengineering, business enterprise architecture compliance, and valid requirements and a viable plan to implement them. Specifically, the guidance states that Air Force core defense business systems are required to comply with the business process reengineering guidance prescribed in the DCMO’s February 2015 Defense Business Systems Investment Management Process Guidance and for systems to assert compliance with the architecture through DCMO’s Integrated Business Framework—Data Alignment Portal. In addition, the guidance states that the department must follow DOD Instruction 5000.75, which requires that certifying officials determine that business requirements are valid and capability efforts have feasible implementation plans.

However, the Air Force guidance does not address the remaining two certification requirements. Officials in the office of the Air Force DCMO acknowledged that the Air Force’s business system certification guidance does not address determining how the acquisition strategy is designed to eliminate or reduce the need to tailor commercial off-the-shelf systems to meet unique requirements, incorporate unique requirements, or incorporate unique interfaces to the maximum extent practicable or is in compliance with DOD’s auditability requirements.

In May 2017, Air Force DCMO officials stated that the department was in the process of developing guidance. However, as of December 2017, the Air Force had not described specific plans to update its business system certification guidance.

Navy. The Department of the Navy issued guidance in May 2016.

This guidance addresses the requirements that a system be certified on the basis of sufficient business process reengineering and business enterprise architecture compliance. In this regard, the guidance provides guidelines for documenting business process reengineering and requires verification that business process reengineering is complete. The guidance also specifies that defense business systems are to map alignment with the business enterprise architecture in DCMO’s Integrated Business Framework–Data Alignment Portal.

Navy’s guidance partially addresses the certification requirement for determining if a defense business system has valid requirements and a viable plan to implement them. Specifically, the guidance includes information on validating requirements; however, it does not include information on determining if a system has a viable plan to implement the requirements.

In addition, Navy’s guidance does not address the remaining two certification requirements, which are to determine that the covered defense business system has an acquisition strategy that eliminates or reduces the need to tailor commercial-off-the-shelf systems, and that the system is in compliance with DOD’s auditability requirements. In August 2017, officials in the Office of the Under Secretary of the Navy (Management) stated that the office was in the process of updating its May 2016 Defense Business System Investment Certification Manual. The officials stated that the goal is to issue interim investment certification guidance by May 2018. As of November 2017, however, Navy had not established a plan for when it expects to publish finalized certification guidance.

Army. The Department of the Army has not issued guidance that addresses any of the act’s certification requirements. The Army issued a template that was to be used to develop fiscal year 2018 portfolio review submissions. However, the template does not address any of the certification requirements.

Officials in the Army’s Office of Business Transformation explained that the Army used DOD DCMO’s 2014 guidance to certify its business systems for fiscal year 2017. In May 2017, they stated that the Army was in the process of developing guidance to implement DOD’s new instruction. In November 2017, an official in the Army’s Office of Business Transformation stated that the office was in the process of completing the guidance and aimed to provide it to the Deputy Under Secretary’s office for signature in January 2018. However, the department has not committed to a specific time frame for when the new guidance is expected to be issued.

Without guidance for the certification authority to determine that defense business systems have addressed each of the act’s certification requirements, the Air Force, Navy, and Army risk allowing systems to proceed into development or production that do not meet these requirements. In particular, the military departments risk wasting funds on developing and maintaining systems that do not have valid requirements and a viable plan to implement the requirements, introduce unnecessary complexity, or that do not adequately support the Department of Defense’s efforts to meet its auditability requirements.

DOD Has Efforts Underway to Improve Its Business Enterprise Architecture, but Its IT Architecture Is Not Complete

According to the NDAA for Fiscal Year 2016, DOD is to develop and maintain a defense business enterprise architecture to guide the development of integrated business processes within the department. In addition, the act states that the business architecture must be consistent with the policies and procedures established by the Director of the Office of Management and Budget. Among other things, OMB policy calls for agencies to develop an enterprise architecture that describes the current architecture, target architecture, and a transition plan to get to the target architecture. The act also calls for the business architecture to contain specific content, including policies, procedures, business data standards, business information requirements, and business performance measures that are to apply uniformly throughout the department.

DOD has developed a business enterprise architecture that is intended to help guide the development of its defense business systems. The department issued version 10 of the business architecture, which is currently being used to support system certification decisions, in February 2013. The business architecture and related documentation include content describing aspects of the current architecture, target architecture, and a transition plan to get to the target architecture. In addition, the business architecture includes content that addresses the act’s requirements. Table 5 provides examples of required content in DOD’s business enterprise architecture.

Nevertheless, some content included in version 10 of the business architecture is outdated and incomplete. For example, version 10 of the business architecture’s repository of laws, regulations, and policies was last updated in February 2013, and officials in the Office of the DOD CIO and Office of the DCMO confirmed that they are not current. Further, the department’s March 2017 business architecture compliance guidance stated that not all relevant business data standards are identified in the business architecture. In addition, based on our review, information about performance measures documented in the architecture is incomplete. For example, target values for performance measures associated with acquisition and logistics initiatives are not identified.

According to officials in the Office of the DCMO, the department is working to update the business architecture. Specifically, the department has developed version 11 of the business architecture to, in part, replace outdated architecture content. According to the officials, version 11 of the architecture is currently available online, but version 10 remains the official version of the business enterprise architecture used for system certification decisions. The officials stated that the department continues to add content to version 11, and they expect that it will be used as the basis of system certification decisions for fiscal year 2019.

In addition, DOD has ongoing work to address a key recommendation we made in July 2015 associated with improving the usefulness of its business architecture. In particular, we reported that the majority of military department portfolio managers that we surveyed believed that the business architecture had not been effective in meeting intended outcomes. For example, only 25 percent of the survey respondents reported that the business architecture effectively enabled DOD to routinely produce timely, accurate, and reliable business and financial information for management purposes. In addition, only 38 percent reported that the business architecture effectively guided the implementation of interoperable defense business systems. As a result, we reported that the architecture had produced limited value and recommended that the department use the results of our survey to determine additional actions that can improve the department’s management of its business enterprise architecture activities.

In response to our recommendation, DOD identified opportunities to address our survey findings and developed a plan for improving its ability to achieve architecture-related outcomes. DOD’s business enterprise architecture improvement plan was signed by the Assistant DCMO in January 2017. However, the department has not yet demonstrated that it has delivered the capabilities described by the plan; thus, we will continue to monitor DOD’s progress to fully address this recommendation.

DOD Has Taken Steps to Develop Its IT Enterprise Architecture, but Does Not Have a Plan That Provides a Road Map for Improving the Department’s IT and Computing Infrastructure

In addition to the business enterprise architecture, according to the act, the DOD CIO is to develop an IT enterprise architecture. This architecture is to describe a plan for improving the IT and computing infrastructure of the department, including for each of the major business processes.

Officials in the Office of the DOD CIO stated that the department considers its information enterprise architecture to be its IT enterprise architecture. The DOD CIO approved version 2.0 of its information enterprise architecture in August 2012. According to DOD documentation, this architecture describes the department’s current information enterprise (i.e., information resources, assets, and processes used to share information across the department and with its mission partners) and includes a vision for the target information enterprise; documents required capabilities, and the activities, rules, and services needed to provide them; and includes information for applying and complying with the architecture.

Nevertheless, while the architecture includes content describing the department’s current and target information enterprise, which is consistent with OMB guidance, it does not include a transition plan that provides a road map for improving the department’s IT and computing infrastructure. Related to this finding, DCMO officials did not agree with our assessment concerning the department’s IT enterprise architecture transition plan. In this regard, officials in the Office of the DCMO stated that the department’s DOD IT Portfolio Repository includes information for managing efforts to improve IT and computing infrastructure at the system level. According to the repository’s data dictionary, this information can include system life cycle start and end dates, as well as information that supports planning for a target environment.

However, documentation describing DOD’s information enterprise architecture does not identify the DOD IT Portfolio Repository as being part of the architecture. Moreover, it does not include a plan for improving the department’s IT and computing infrastructure for each of the major business processes. Officials in the Office of the CIO acknowledged that the architecture does not include such plans. According to the officials, the department is currently developing version 3.0 of its information enterprise architecture (i.e., its IT enterprise architecture). The officials stated that the department does not currently intend for the architecture to include a plan for improving the department’s IT and computing infrastructure that addresses each of the major business processes. They added, however, that there is an effort to ensure that functional areas, such as human resources management, are included. DCMO officials stated that the department has not defined how the DOD IT enterprise architecture needs to be segmented for each major business process because the infrastructure requirements seem to be similar for each of the processes.

Without an architecture that includes a plan for improving its IT and computing infrastructure, including for each of the major business processes, DOD risks not ensuring that stakeholders across the department have a consistent understanding of the steps needed to achieve the department’s future vision, agency priorities, potential dependencies among investments, and emerging and available technological opportunities.

DOD Has Not Demonstrated That Its Business and IT Architectures Are Integrated

According to the act, the DOD business enterprise architecture is to be integrated into the DOD IT enterprise architecture. The department’s business architecture compliance guide also recognizes that the business architecture is to be integrated with the IT enterprise architecture.

However, the department has not demonstrated that it has integrated the business enterprise architecture into the information enterprise architecture. Specifically, the department did not provide documentation associated with either architecture that describes how the two are, or are to be, integrated. The business enterprise architecture compliance guide states that DOD Directive 8000.01 implements the requirement that the two architectures are to be integrated. However, the directive does not address how they are, or are to be, integrated.

Officials in the Offices of the CIO and the DCMO described steps they were taking to coordinate the development of the next versions of the information enterprise architecture (i.e., IT enterprise architecture) and business enterprise architecture. However, these steps were not sufficient to help ensure integration of the two architectures. Specifically, in June 2017, officials in the Office of the DOD CIO stated they were participating in the development of the next version of the business architecture and that the DOD CIO is represented on the Business Enterprise Architecture Configuration Control Board. Officials in the Office of the DCMO confirmed that DOD CIO officials participate on the board. However, officials from the Office of the DCMO said that, until it met in June 2017 the board had not met since 2014. Moreover, documentation of the June 2017 meeting, and a subsequent November 2017 meeting, did not indicate that the board members had discussed integration of the department’s business and information enterprise architectures.

In addition, officials in the Office of the DCMO reported that the office has not actively participated in the information enterprise architecture working group. Further, our review of meeting minutes from this working group did not identify participation by officials in the Office of the DCMO, or that integration of the architectures was discussed. The Office of the DCMO described other mechanisms for its sharing of information about architectures with the Office of the DOD CIO. For example, the Office of the DCMO stated that it participates with DOD CIO bodies governing version 3.0 development. Nevertheless, the Office of the DCMO reiterated that technical integration of the architectures has not been designed.

Until DOD ensures that its business architecture is integrated into its IT enterprise architecture, the department may not be able to ensure that its business strategies capitalize on technologies and that its IT infrastructure will support DOD’s business priorities and related business strategies.

The Defense Business Council Addressed Legislative Provisions to Provide Advice on Defense Business Systems to the Secretary of Defense

The NDAA for Fiscal Year 2016 requires the Secretary to establish a Defense Business Council, chaired by the DCMO and the DOD CIO, to provide advice to the Secretary on: developing the business enterprise architecture, reengineering the department’s business processes, developing and deploying business systems, and developing requirements for business systems.

DOD established the department’s Defense Business Council in October 2012, prior to the act. According to its current charter, dated December 2014, the Council is co-chaired by the DCMO and the DOD CIO. In addition, the Council is to serve as the principal governance body for vetting issues related to managing and improving defense business operations. Among other things, it serves as the investment review board for defense business system investments.

The Defense Business Council charter also states that the Council was established as a principal supporting tier of governance to the Deputy’s Management Action Group. The Deputy’s Management Action Group was established by an October 2011 memorandum issued by the Deputy Secretary of Defense. According to information published on DCMO’s website, the group was established to be the primary civilian-military management forum that supports the Secretary of Defense, and is to address top department issues that have resource, management, and broad strategic and/or policy implications. The group’s primary mission is to produce advice for the Deputy Secretary of Defense in a collaborative environment and to ensure that the group’s execution aligns with the Secretary of Defense’s priorities. According to the Office of the DCMO, the Defense Business Council determines whether or not to elevate a topic to the Deputy’s Management Action Group to address on behalf of the Secretary.

Based on our review of meeting documentation for 27 meetings that the Defense Business Council held between January 2016 and August 2017, the Council discussed the four topics on which the NDAA for Fiscal Year 2016 requires it to provide advice to the Secretary. According to the Office of the DCMO, during the discussions of these topics, the Council did not identify any issues related to the topics that needed to be elevated to the Deputy’s Management Action Group. Table 6 identifies the number of meetings in which the Council discussed each topic during this time period.

By ensuring that the required business system topics are discussed during Defense Business Council meetings, the department should be positioned to raise issues to the Deputy’s Management Action Group, and ultimately, to advise the Secretary of Defense on matters associated with these topics.

DOD Certified Selected Business Systems for Fiscal Year 2017 on the Basis of Earlier Certification Requirements

The NDAA for Fiscal Year 2016 requires that, for any fiscal year in which funds are expended for development or sustainment pursuant to a covered defense business system program, the Secretary of Defense is to ensure that a covered business system not proceed into development (or, if no development is required, into production or fielding) unless the appropriate approval official reviews the system to determine if the system meets five key requirements, as previously discussed in this report. In addition, the act requires that the appropriate approval official certify, certify with conditions, or decline to certify that the system satisfies these five requirements.

The department issued DOD Instruction 5000.75, which established business system categories and assigned certifying officials, consistent with the act. Table 7 describes the business system categories and the assigned certifying officials, as defined in DOD Instruction 5000.75.

The DOD DCMO certified the five systems in our sample (which included the military departments’ systems) for fiscal year 2017. However, these certifications were issued in accordance with the previous fiscal year’s (fiscal year 2016) certification requirements. Those requirements had stipulated that a defense business system program was to be reviewed and certified on the basis of the system’s compliance with the business enterprise architecture and appropriate business process reengineering, rather than on the basis of having met all five requirements identified in the NDAA for Fiscal Year 2016.

Specifically, DCMO certified the systems on the basis of determining that the systems were in compliance with the business enterprise architecture and had been sufficiently reengineered. However, none of the systems were certified on the basis of a determination that they had valid, achievable requirements and a viable plan for implementing them; had an acquisition strategy to reduce or eliminate the need to tailor commercial off-the-shelf systems; or were in compliance with the department’s auditability requirements.

Officials in the Offices of the DOD DCMO, the Air Force DCMO, the Under Secretary of the Navy (Management), and Army Business Transformation told us that the systems were not certified relative to three of the requirements because the department did not issue guidance to reflect changes made by the NDAA for Fiscal Year 2016 in time for the fiscal year 2017 certification process. Prior to the NDAA for Fiscal Year 2016, relevant legislation and DOD guidance only called for annual determinations to be made regarding whether a system complied with the business enterprise architecture and whether appropriate business process reengineering had been conducted.

In January 2016, the DCMO issued a memorandum stating that the department planned to issue new guidance and policy to implement the new legislation by the end of February 2016. However, the department did not issue additional guidance addressing the new certification requirements until April 2017. The system certifications, which were required by the act to be completed before systems could spend fiscal year 2017 funds, occurred in August and September 2016.

In explaining the delay in issuing new guidance on the certification requirements, officials in the Office of the DCMO stated that the statutory deadline for issuing guidance was December 31, 2016. They added that, given this statutory deadline, and the start of fiscal year 2017 on October 1, 2016, it was their determination that Congress did not intend for the NDAA for Fiscal Year 2016’s certification requirements to be fully implemented before fiscal year 2017 started. DCMO officials stated that they intend for the department to use the certification requirements established by the NDAA for Fiscal Year 2016 for future system certifications.

While it was reasonable for the department to use the earlier guidance for its fiscal year 2017 certifications, given that the new guidance had not yet been issued, it will be important going forward that the department certifies business systems on the basis of the certification requirements established in the NDAA for Fiscal Year 2016 and its related guidance addressing these requirements. Certifying systems on the basis of the act’s requirements should help ensure that funds are not wasted on developing and maintaining systems that do not have valid requirements and a viable plan to implement the requirements, that introduce unnecessary complexity, or that impede the Department of Defense’s efforts to meet its auditability requirements.

Conclusions

Since the NDAA for Fiscal Year 2016 was signed in November 2015, DOD has issued guidance that addresses most provisions of the NDAA for Fiscal Year 2016 related to managing defense business system investments. However, the department has not established policies requiring consideration of sustainability and technology requirements and the use of best systems engineering practices in the procurement and deployment of its systems. Having these policies would better enable the department to ensure it is efficiently and effectively procuring and deploying its business systems.

In addition, the Air Force, the Army, and Navy have made mixed progress in issuing guidance to assist in making certification decisions regarding systems within their respective areas of responsibility. Specifically, the Air Force and Navy issued guidance on the certification of business systems that does not fully address new certification requirements, while the Army has not issued any updated guidance for its certifications. As a result, the Air Force, Navy, and Army risk wasting funds on developing and maintaining systems that do not have valid requirements and a viable plan to implement the requirements, introduce unnecessary complexity, or do not adequately support the Department of Defense’s efforts to meet its auditability requirements.

Also, DOD has developed an IT architecture, but this architecture does not address the act’s requirement that it include a plan for improving the department’s IT and computing infrastructure, including for each business process. In addition, DOD’s plans for updating its IT architecture do not address how the department intends to integrate its business and IT architectures, as called for by the act. As a result, DOD risks not having a consistent understanding of what is needed to achieve the department’s future vision, agency priorities, potential dependencies among investments, and emerging and available technological opportunities.

Recommendations for Executive Action

We are making six recommendations, including three to the Secretary of Defense and one to each of the Secretaries of the Air Force, the Navy and the Army: The Secretary of Defense should define a specific time frame for finalizing, and ensure the issuance of (1) policy requiring full consideration of sustainability and technological refreshment requirements for its defense business system investments; and (2) policy requiring that best systems engineering practices are used in the procurement and deployment of commercial systems, modified commercial systems, and defense-unique systems to meet DOD missions. (Recommendation 1)

The Secretary of the Air Force should define a specific time frame for finalizing, and ensure the issuance of guidance for certifying the department’s business systems on the basis of (1) having an acquisition strategy designed to eliminate or reduce the need to tailor commercial off- the-shelf systems to meet unique requirements, incorporate unique requirements, or incorporate unique interfaces to the maximum extent practicable; and (2) being in compliance with DOD’s auditability requirements. (Recommendation 2)

The Secretary of the Navy should define a specific time frame for finalizing, and ensure the issuance of guidance for certifying the department’s business systems on the basis of (1) having a viable plan to implement the system’s requirements; (2) having an acquisition strategy designed to eliminate or reduce the need to tailor commercial off-the-shelf systems to meet unique requirements, incorporate unique requirements, or incorporate unique interfaces to the maximum extent practicable; and (3) being in compliance with DOD’s auditability requirements. (Recommendation 3)

The Secretary of the Army should define a specific time frame for finalizing, and ensure the issuance of guidance for certifying the department’s business systems on the basis of (1) being reengineered to be as streamlined and efficient as practicable, and determining that implementation of the system will maximize the elimination of unique software requirements and unique interfaces; (2) being in compliance with the business enterprise architecture; (3) having valid, achievable requirements and a viable plan to implement the requirements; (4) having an acquisition strategy designed to eliminate or reduce the need to tailor commercial off-the-shelf systems to meet unique requirements, incorporate unique requirements, or incorporate unique interfaces to the maximum extent practicable; and (5) being in compliance with DOD’s auditability requirements. (Recommendation 4)

The Secretary of Defense should ensure that the DOD CIO develops an IT enterprise architecture which includes a transition plan that provides a road map for improving the department’s IT and computing infrastructure, including for each of its business processes. (Recommendation 5)

The Secretary of Defense should ensure that the DOD CIO and Chief Management Officer work together to define a specific time frame for when the department plans to integrate its business and IT architectures and ensure that the architectures are integrated. (Recommendation 6)

Agency Comments and Our Evaluation

DOD provided written comments on a draft of this report, which are reprinted in appendix III. In the comments, the department stated that it concurred with three of the recommendations and partially concurred with three of the recommendations. DOD also provided evidence that it has fully addressed one of the recommendations. In addition, DOD provided technical comments that we incorporated in the report, as appropriate.

DOD stated that it concurred with our first recommendation, which called for it to define a specific time frame for finalizing, and ensure the issuance of, policies that fully address provisions in the NDAA for Fiscal Year 2016. Furthermore, the department stated that it had complied with the recommendation. Specifically, the department stated that it had published its defense business systems investment management guidance in April 2017. This guidance identifies DOD’s Financial Management Regulation, Volume 2B, Chapter 18 “Information Technology” and supporting IT budget policy and guidance as well as DOD Instruction 5000.75 and supporting acquisition policy and guidance. The department stated that the Financial Management Regulation specifically addresses the requirement for sustainability and technological refreshment requirements for its defense business system investments.

While DOD reported taking this action, we do not agree that the department has complied with our recommendation. In reviewing the department’s guidance, we found that none of the cited management documents includes a policy requiring consideration of sustainability and technological refreshment requirements for DOD’s defense business systems. Further, none of these documents includes a policy requiring that best systems engineering practices be used in the procurement and deployment of commercial, modified-commercial, and defense unique systems.

Without a policy requiring full consideration of sustainability and technological refreshment requirements for its defense business system investments, the department may not be able to ensure that it has a full understanding of the costs associated with these requirements. Further, without a policy requiring the use of best systems engineering practices in systems procurement and deployment, the department may be limited in its ability to effectively balance meeting system cost and performance objectives. Accordingly, we continue to believe that our recommendation is valid.

The department concurred with our second recommendation, that the Secretary of the Air Force define a specific time frame for finalizing, and ensure the issuance of, guidance that fully addresses certification requirements, in accordance with the NDAA for Fiscal Year 2016. Moreover, the department stated that the Air Force has complied with the recommendation. Specifically, DOD stated that Air Force Manual 63-144 details the consideration of using existing commercial solutions without modification or tailoring.

However, while the manual provides a foundation on which the Air Force can build, it is not sufficient to fully address our recommendation because it does not include guidance on certifying business systems on the basis of having an acquisition strategy that eliminates or reduces the need to tailor commercial-off-the-shelf systems. In addition, the department did not demonstrate that the Air Force has issued guidance for certifying business systems on the basis of being in compliance with DOD’s auditability requirements. Rather, the Air Force stated that it has pending guidance that addresses the acquisition strategy and auditability requirements. We plan to evaluate the guidance to determine the extent to which it addresses our recommendation after it is issued.

The department partially agreed with our third recommendation, that the Secretary of the Navy define a specific time frame for finalizing, and ensure the issuance of, guidance that fully addresses certification requirements. Specifically, DOD stated that Navy agreed to issue guidance. Subsequently, on March 8, 2018, Navy issued its updated guidance.

However, Navy disagreed with the recommendation, as written, and suggested that GAO revise the recommendation to state that “The Secretary of the Navy should ensure guidance is issued according to established timeline for certifying the department’s business systems. . .” According to Navy, this change would support alignment with the timeline for certifying the department’s business systems driven by the Chief Management Officer investment review timeline.

Based on our analysis, we found the guidance that Navy issued to be consistent with our recommendation. Thus, we plan to close the recommendation as fully implemented. We have also annotated this report, where appropriate, to explain that the Navy issued guidance while the draft of this report was at the department for comment. On the other hand, we did not revise the wording of our recommendation, as we believe it appropriately reflected the importance of Navy taking action to ensure the issuance of its guidance.

The department stated that it concurred with our fourth recommendation, which called for the Secretary of the Army to define a specific time frame for finalizing, and ensure the issuance of, guidance for certifying the department’s business systems on the basis of the certification requirements. Furthermore, on March 23, 2018, the Army issued its guidance. However, because of the timing of this report relative to when the Army provided its guidance to us (on March 27, 2018), we have not yet completed an assessment of the guidance. We have annotated this report, where appropriate, to reflect the Army’s action on our recommendation.

The department stated that it partially concurred with our fifth recommendation. This recommendation called for the DOD CIO to develop an IT enterprise architecture which includes a transition plan that provides a road map for improving the department’s IT and computing infrastructure, including for each of its business processes. Toward this end, the department agreed that the DOD CIO should develop an architecture that enables improving the department’s IT and computing infrastructure for each of its business processes. However, the department also stated that the recommendation is not needed because the goal is already being accomplished by a set of processes, organizations, protocols, and architecture data.

For example, the department described processes and relationships between the Office of the DOD CIO and the Office of the Chief Management Officer and the boards that support the department’s business and IT enterprise architectures. In particular, the department stated that information enterprise architecture data relevant to the business enterprise are accessed via the DOD Information Enterprise Architecture Data Selection Wizard and imported into the business enterprise architecture. The department further stated that, if the business capability acquisition cycle process indicates a need to improve the IT or computing infrastructure, the Office of the Chief Management Officer has a protocol to initiate a proposal to change the information enterprise architecture.

We agree that the department’s processes, organizations, protocols, and architecture data are keys to successful IT management. However, during the course of our audit, we found that documentation describing DOD’s IT architecture did not include a plan for improving the department’s IT and computing infrastructure for each of the major business processes. Moreover, officials in the Office of the CIO acknowledged that the architecture did not include such a plan.

Without a transition plan that provides a road map for improving the department’s IT and computing infrastructure, including for each of its business processes, it will be difficult for the department to rely on its personnel to timely and proactively manage and direct modernization efforts of such a magnitude as DOD’s systems modernization efforts. Further, without such a plan, DOD risks not being able to ensure that stakeholders across the department have a consistent understanding of the steps needed to achieve the department’s future vision, agency priorities, potential dependencies among investments, and emerging and available technological opportunities. Thus, we maintain that the department should fully implement our recommendation.

The department stated that it partially concurred with our sixth recommendation, that the DOD CIO and DCMO work together to define a specific time frame for when the department plans to integrate its business and IT architectures. In particular, the department stated that it agrees that the DOD CIO and Chief Management Officer should work together to establish a time frame and ensure coordination and consistency of the IT and business architectures. However, the department disagreed with the use and intent of the term “integrate,” as stated in the recommendation, although it did not explain the reason for this disagreement. Instead, it proposed that we change our recommendation to read “The GAO recommends the Secretary of Defense ensure the DoD CIO and CMO work together to define a specific timeline for coordinating its business and IT architectures to achieve better enterprise alignment among the architectures.”

We agree that it is important to achieve coordination and consistency between the business and IT architectures. However, the department did not provide documentation associated with either architecture that describes how the two are, or are to be, integrated, as called for by the NDAA for Fiscal Year 2016 and DOD guidance.

Integrating the architectures would help ensure that business strategies better capitalize on existing and planned technologies and that IT solutions and infrastructure support business priorities and related business strategies. Thus, we continue to believe that our recommendation is valid. However, we have updated the recommendation to state that the DOD CIO and the Chief Management Officer should work together. We made this change because, effective February 1, 2018, the Secretary of Defense eliminated the DCMO position and expanded the role of the Chief Management Officer, in accordance with the National Defense Authorization Act for Fiscal Year 2018.

We are sending copies of this report to appropriate congressional committees; the Secretary of Defense; the Secretaries of the Army, Navy, and Air Force; and the Director of the Office of Management and Budget. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff members have any questions about this report, please contact me at (202) 512-4456 or harriscc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made contributions to this report are listed in appendix IV.

Appendix I: Objective, Scope, and Methodology

Our objective was to determine the actions taken by the Department of Defense (DOD) to comply with provisions included in the National Defense Authorization Act for Fiscal Year 2016 (NDAA). These provisions require DOD to perform certain activities aimed at ensuring that its business system investments are managed efficiently and effectively. Specifically, we determined to what extent DOD has 1. established guidance for effectively managing its defense business 2. developed and maintained a defense business enterprise architecture and information technology (IT) enterprise architecture, in accordance with relevant laws and Office of Management and Budget (OMB) policies and guidance; 3. used the Defense Business Council to provide advice to the Secretary on developing the business enterprise architecture, reengineering the department’s business processes, developing and deploying business systems, and developing requirements for business systems; and 4. ensured that covered business systems are reviewed and certified in accordance with the act.

To address the extent to which DOD has established guidance for effectively managing defense business system investments, we obtained and analyzed the department’s guidance, as well as the guidance established by the Departments of the Air Force, Army, and Navy, for managing defense business systems relative to the act’s requirements. Specifically, the NDAA for Fiscal Year 2016 required the Secretary of Defense to issue guidance, by December 31, 2016, to provide for the coordination of and decision making for the planning, programming, and control of investments in covered defense business systems. The act required this guidance to include the following six elements:

Policy to ensure DOD business processes are continuously reviewed and revised to implement the most streamlined and efficient business processes practicable and eliminate or reduce the need to tailor commercial off-the-shelf systems to meet or incorporate requirements or interfaces that are unique to the department.

Process to establish requirements for covered defense business systems.

Mechanisms for planning and controlling investments in covered defense business systems, including a process for the collection and review of programming and budgeting information for covered defense business systems.

Policy requiring the periodic review of covered defense business systems that have been fully deployed, by portfolio, to ensure that investments in such portfolios are appropriate.

Policy to ensure full consideration of sustainability and technological refreshment requirements, and the appropriate use of open architectures.

Policy to ensure that best acquisition and systems engineering practices are used in the procurement and deployment of commercial systems, modified commercial systems, and defense-unique systems to meet DOD missions.

We assessed the February 2017 DOD Instruction 5000.75, Business Systems Requirements and Acquisitions, and April 2017 defense business system investment management guidance, which the department issued to address the act’s requirements. In addition, we assessed the department’s Financial Management Regulation and directive on its planning, programming, budgeting, and execution process, which the department stated also address the act’s provisions.

We also assessed DOD’s guidance for managing business system investments relative to the act’s business system certification requirements. The act requires that the Secretary of Defense ensure that a covered defense business system not proceed into development (or, if no development is required, into production or fielding) unless the appropriate approval official determines that the system meets five requirements. The act further requires for any fiscal year in which funds are expended for development or sustainment pursuant to a covered defense business system program, the appropriate approval official to review the system to determine if the system: has been, or is being, reengineered to be as streamlined and efficient as practicable, and whether the implementation of the system will maximize the elimination of unique software requirements and unique interfaces; is in compliance with the business enterprise architecture or will be in compliance as a result of planned modifications; has valid, achievable requirements, and a viable plan for implementing those requirements (including, as appropriate, market research, business process reengineering, and prototyping activities); has an acquisition strategy designed to eliminate or reduce the need to tailor commercial off-the-shelf systems to meet unique requirements, incorporate unique requirements, or incorporate unique interfaces to the maximum extent practicable; and is in compliance with the department’s auditability requirements.

We compared Office of the Deputy Chief Management Office (DCMO) certification guidance with the act’s certification requirements. In addition, we compared the guidance established by the Departments of the Air Force, the Army, and the Navy for certifying their business systems with the act’s certification requirements.

We also interviewed cognizant officials responsible for managing defense business system investments at DOD, including the military departments. Specifically, we interviewed officials in the Office of the DCMO, the Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics, the Office of the Chief Information Officer (CIO), and the Offices of the CMOs in the Departments of the Air Force, Army, and Navy.

To determine the extent to which DOD has developed and maintained a defense business enterprise architecture and IT enterprise architecture, in accordance with relevant laws and OMB policy and guidance, we assessed the business enterprise architecture against the relevant laws and OMB policy and guidance; the IT enterprise architecture against the relevant laws and OMB policy and guidance; and the department’s efforts to integrate its business and IT architectures against the act’s requirement.

To determine the extent to which the department has developed and maintained a business enterprise architecture in accordance with relevant laws and OMB policy and guidance, we reviewed version 10 of its business enterprise architecture, which was released in February 2013, and related information relative to the act’s requirements; U.S. Code, Title 44, Section 3601, which defines an enterprise architecture; and OMB policy and guidance. We also reviewed version 11 of the architecture to determine the extent to which it differed from version 10.

Further, we reviewed the department’s business enterprise architecture improvement plan, which it developed in response to a recommendation we made in July 2015. Specifically, we recommended that the department use the results of our portfolio manager survey to determine additional actions that could improve the department’s management of its enterprise architecture activities. In response to our recommendation, the department developed and approved a plan in January 2017. We assessed the extent to which the department had delivered the planned capabilities relative to the plan. We also reviewed the extent to which the delivery dates of the three planned capabilities and associated tasks changed over time relative to the plan.

To assess the extent to which the department developed and maintained an IT enterprise architecture in accordance with relevant laws and OMB policy and guidance, we reviewed content from the department’s IT enterprise architecture and compared it with requirements from the act, U.S. Code, Title 44, Section 3601, and OMB policy and guidance.

Specifically, we reviewed version 2.0 of the department’s information enterprise architecture, which was released in August 2012, relative to the act’s requirement for the DOD CIO to develop an IT enterprise architecture that is to describe a plan for improving the IT and computing infrastructure of the department, including for each of the major business processes. We reviewed volumes I and II of the information enterprise architecture and the four enterprise-wide reference architectures to determine if the architecture described a plan for improving the IT and computing infrastructure of the department, as called for by the act. We also reviewed whether the architecture included content that described the current and the target environments, and a transition plan to get from the current to the target environment, consistent with OMB policy and guidance.

To determine the extent to which the department has integrated its business and IT architectures, as required by the act, we reviewed DOD Directive 8000.01, Management of the Department of Defense Information Enterprise. We also reviewed meeting documentation from the information enterprise architecture working group responsible for the development of an updated architecture. In addition, we reviewed meeting documentation from the Business Enterprise Architecture Configuration Control Board to identify any discussions among CIO and DCMO officials regarding integration of the two architectures, as well as the level of participation by both parties.

Finally, we interviewed officials in the Office of the DCMO and the Office of the CIO about efforts to develop and maintain a business enterprise architecture, develop an IT enterprise architecture, and integrate the business and IT architectures.

To determine the extent to which the department has used the Defense Business Council to provide advice to the Secretary of Defense on developing the business enterprise architecture, reengineering the department’s business processes, developing and deploying business systems, and developing requirements for business systems, in accordance with the act, we analyzed the department’s December 2014 Defense Business Council Charter and April 2017 defense business systems investment management guidance. We compared information in the charter and guidance to the requirement that the Secretary establish the Defense Business Council to advise the Secretary on the required defense business system topics.

In addition, we obtained and analyzed meeting summaries and briefings for 27 Defense Business Council meetings that took place from January 2016 through August 2017. Specifically, we assessed the frequency with which the meetings held during this time period addressed the required topics. We chose this time period because 2016 was the first calendar year following the enactment of the NDAA for Fiscal Year 2016. Further, we chose August 2017 as our end date because it was the last month’s data that we could reasonably expect to obtain and review within our reporting time frame. We also interviewed officials in the Offices of the DCMO and CIO about the Defense Business Council and the Deputy’s Management Action Group, which is the governance entity to which the Council reports.

To determine the extent to which DOD has ensured that covered business systems are reviewed and certified in accordance with the act, we reviewed a nongeneralizable sample of business systems from DOD’s two categories of covered defense business systems that require certification. To select the sample, we considered Category I systems, which were systems that were expected to have a total amount of budget authority of more than $250 million over the period of the current future- years defense program, and Category II systems, which were systems that were expected to have a total amount of budget authority of between $50 million and $250 million over the period of the future-years defense program.

We further categorized the Category II systems into four groups—those owned by the Air Force, the Army, Navy, and the remaining DOD components. We selected one system with the highest expected cost over the course of the department’s future-years defense program from each group. This resulted in our selection of five systems: one Category I system, one Category II system from each military department, and one Category II system from the remaining DOD components.

We reviewed, respectively, DOD’s Healthcare Management System Modernization Program; Air Force’s Maintenance, Repair and Overhaul initiative; Army’s Reserve Component Automation System; Navy’s Electronic Procurement System; and the Defense Logistics Agency’s Defense Agencies Initiative Increment 2. We determined that the number of systems we selected was sufficient for our evaluation.

For each system, we assessed the extent to which it had been certified on the basis of the five certification requirements in the act. Specifically, we evaluated investment decision memos and certification assertions to determine if each system had been certified according to the act’s requirements, which include ensuring that the system had been, or was being, reengineered to be as streamlined and efficient as practicable, and the implementation of the system would maximize the elimination of unique software requirements and unique interfaces; was in compliance with the business enterprise architecture or would be in compliance as a result of planned modifications; had valid, achievable requirements, and a viable plan for implementing those requirements; had an acquisition strategy designed to eliminate or reduce the need to tailor commercial off- the-shelf systems to meet unique requirements, incorporate unique requirements, or incorporate unique interfaces to the maximum extent practicable; and was in compliance with the department’s auditability requirements. We did not determine whether the certification assertions were valid. For example, we did not evaluate business process reengineering activities to determine if they were sufficient. We also interviewed DOD DCMO and military department officials about the certification of these systems.

To determine the reliability of the business system cost data used to select the systems, we reviewed system documentation for the three systems DOD uses to store data, which include the Defense Information Technology Investment Portal, the DOD Information Technology Portfolio Repository, and the Select and Native Programming-Information Technology system. In this regard, we requested and reviewed department responses to questions about the systems and about how the department ensures the quality and reliability of the data. In addition, we requested and reviewed documentation related to the systems (e.g., data dictionaries, system instructions, and user training manuals) and reviewed the data for obvious issues, including missing or questionable values. We also reviewed available reports on the quality of the inventories (e.g., inspector general reports). We found the data to be sufficiently reliable for our purpose of selecting systems for evaluation.

We conducted this performance audit from January 2017 to March 2018 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Prior GAO Reports on Department of Defense Business System Modernization

Since 2005, we have issued 11 reports assessing DOD’s actions to respond to business system modernization provisions contained in U.S. Code, Title 10, Section 2222. The reports are listed below.

DOD Business Systems Modernization: Additional Action Needed to Achieve Intended Outcomes, GAO-15-627 (Washington, D.C.: July 16, 2015).

Defense Business Systems: Further Refinements Needed to Guide the Investment Management Process, GAO-14-486 (Washington, D.C. May 12, 2014).

DOD Business Systems Modernization: Further Actions Needed to Address Challenges and Improve Accountability, GAO-13-557 (Washington, D.C.: May 17, 2013).

DOD Business Systems Modernization: Governance Mechanisms for Implementing Management Controls Need to Be Improved, GAO-12-685 (Washington, D.C.: June 1, 2012).

Department of Defense: Further Actions Needed to Institutionalize Key Business System Modernization Management Controls, GAO-11-684 (Washington, D.C.: June 29, 2011).

Business Systems Modernization: Scope and Content of DOD’s Congressional Report and Executive Oversight of Investments Need to Improve, GAO-10-663 (Washington, D.C.: May 24, 2010).

DOD Business Systems Modernization: Recent Slowdown in Institutionalizing Key Management Controls Needs to Be Addressed, GAO-09-586 (Washington, D.C.: May 18, 2009).

DOD Business Systems Modernization: Progress in Establishing Corporate Management Controls Needs to Be Replicated Within Military Departments, GAO-08-705 (Washington, D.C.: May 15, 2008).

DOD Business Systems Modernization: Progress Continues to Be Made in Establishing Corporate Management Controls, but Further Steps Are Needed, GAO-07-733 (Washington, D.C.: May 14, 2007).

Business Systems Modernization: DOD Continues to Improve Institutional Approach, but Further Steps Needed, GAO-06-658 (Washington, D.C.: May 15, 2006).

DOD Business Systems Modernization: Important Progress Made in Establishing Foundational Architecture Products and Investment Management Practices, but Much Work Remains, GAO-06-219 (Washington, D.C.: November 23, 2005).

Appendix III: Comments from the Department of Defense

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact above, individuals making contributions to this report include Michael Holland (Assistant Director), Cheryl Dottermusch (Analyst in Charge), John Bailey, Chris Businsky, Camille Chaires, Nancy Glover, James Houtz, Anh Le, Tyler Mountjoy, Monica Perez-Nelson, Priscilla Smith, and Adam Vodraska.