Defense Business Systems: Further Refinements Needed to Guide the Investment Management Process
Published: 2014-05-12T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: GAO designated DOD's multibillion-dollar business systems modernization program as high risk in 1995, and since then has provided a series of recommendations aimed at strengthening DOD's institutional approach to modernizing its business systems investments. Section 332 of the Fiscal Year 2005 NDAA requires the department to take specific actions relative to its modernization efforts and GAO to assess actions taken by DOD to comply with section 332 of the act. In evaluating DOD's compliance, GAO analyzed, among other things, investment management policies and procedures, certification actions for business system investments, and the latest versions of the department's business enterprise architecture and enterprise transition plan.
section_title: What GAO Found
paragraphs: The Department of Defense (DOD) has developed a portfolio-based investment management process for its defense business systems, certified and approved a majority of its defense business systems, made key improvements to the data used to manage its business investments, and updated its transition plan to assist its efforts in complying with key provisions of section 332 of the Ronald W. Reagan National Defense Authorization Act for Fiscal Year 2005 (NDAA or “the act”), as amended (10 U.S.C. § 2222). However, the department continues to face challenges in fully complying with the act's requirements, modernizing its business systems environment, and addressing GAO's prior recommendations (see table).
Department officials cited various reasons for the shortfalls noted above. For example, they stated that aligning the investment review process with the budgeting process takes time to coordinate. They also noted that different systems are reviewed with different levels of scrutiny based on various factors, but those factors are not documented in existing guidance. Continued progress in improving its investment management approach will allow DOD to more effectively manage the billions of dollars the department invests annually in modernizing its business systems.
section_title: What GAO Recommends
paragraphs: GAO is making three recommendations to help improve the department's business system investment management process and business enterprise architecture. DOD concurred with two recommendations and partially concurred with the third—to define criteria for reviewing business systems at appropriate levels in the department. In particular, DOD noted that systems are reviewed within components before being reviewed by the executive-level investment review board. However, until DOD ensures that investments are reviewed at an appropriate level based on defined criteria such as cost, scope, complexity, and risk, the department is at an increased risk of failing to identify and address important issues associated with large-scale and costly systems.

section_title: Letter
paragraphs: For decades, the Department of Defense (DOD) has been challenged in  modernizing its business systems. In 1995, we designated the  department’s business systems modernization program as high risk  because of its vulnerability to fraud, waste, abuse, and mismanagement,  and because of missed opportunities to achieve greater efficiencies; and  we continue to designate it as such today. In addition, we have reported  that significant potential exists for identifying and avoiding costs  associated with duplicative functionality across DOD business system  investments, which include approximately 2,329 systems. DOD’s  business systems include those for personnel, financial management,  healthcare, and logistics, and cost the department billions of dollars each  year.
Since designating this area as high risk in 1995, we have made a series  of recommendations aimed at strengthening DOD’s institutional approach  to modernization and reducing the risk associated with key investments.  Further, Congress included provisions in the Ronald W. Reagan National  Defense Authorization Act for Fiscal Year 2005 (NDAA), as amended,  that were consistent with our recommendations. Specifically, section 332  of the act, as amended, requires the department to, among other things,  (1) establish a systems investment approval and accountability structure  along with an investment review process, (2) certify and approve any  business system program costing in excess of $1 million, (3) develop a  business enterprise architecture (BEA) to cover all defense business  systems, (4) develop a transition plan for implementing the architecture,  and (5) identify systems information in its annual budget submission. The  act directs the Secretary of Defense to submit an annual report to the  congressional defense committees on DOD’s compliance with certain  requirements of the NDAA not later than March 15 of each year, through  2016.
The act also directed us to report to these congressional committees— within 60 days of DOD’s report submission—an assessment of the  department’s actions to comply with the requirements of the act.
Accordingly, our objective was to assess the actions taken by DOD to  comply with section 332 of the act.
To accomplish the objective, we reviewed and analyzed the latest version  of DOD’s investment management policies and procedures, BEA, and  enterprise transition plan (ETP), using our prior reports as a baseline. To  address the investment management provisions of the act, we reviewed  the department’s report to Congress, which was submitted on March 14,  2014, analyzed the most recent investment management guidance, and  compared this information against the law and our information technology  (IT) investment management guidance. To determine the extent to which  DOD has certified and approved business systems, we reviewed  certification and approval memoranda, related documentation, and  associated data, and compared them to the act’s requirements and our  previous recommendations. To assess DOD’s progress in developing the  BEA and ETP, we reviewed the most recent versions of the BEA and  plan, and compared this information against the act’s requirements and  our prior recommendations. To assess the system information in the  budget submission, we reviewed summary budget data prepared for the  fiscal year 2015 submission, and compared them to the requirements in  the act.
We conducted this performance audit from October 2013 to May 2014, in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objective. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objective. Additional details on our  objective, scope, and methodology are contained in appendix I.
subsections: section_title: Background
paragraphs: DOD is one of the largest and most complex organizations in the world. In  support of its military operations, the department performs an assortment  of interrelated and interdependent business functions, such as logistics  management, procurement, health care management, and financial  management. Yet, we have previously reported that the DOD systems  environment that supports these business functions is overly complex and  error prone, and is characterized by (1) little standardization across the  department, (2) multiple systems performing the same tasks, (3) the  same data stored in multiple systems, and (4) the need for data to be  entered manually into multiple systems.
For fiscal year 2015, the department requested about $10.038 billion for  its business system investments. According to the department’s  authoritative source for certification data, its environment is composed of  approximately 2,329 business systems, including 228 for acquisition, 14  for defense security enterprise, 28 for enterprise IT infrastructure, 286 for  financial management, 730 for human resources management, 293 for  installations and environment, 702 for logistics and materiel readiness,  and 8 for security cooperation. (See fig. 1.) Of these 2,329 business  systems, 1,180 are covered by the act’s certification and approval  requirements.
DOD currently bears responsibility, in whole or in part, for 15 of the 30  programs across the federal government that we have designated as high  risk. Seven of these areas are specific to the department, and eight  other high-risk areas are shared with other federal agencies.  Collectively, these high-risk areas relate to DOD’s major business  operations that are inextricably linked to the department’s ability to  perform its overall mission and directly affect the readiness and  capabilities of U.S. military forces and can affect the success of a  mission. In particular, the department’s nonintegrated and duplicative  systems impair its ability to combat fraud, waste, and abuse.  Consequently, DOD’s business systems modernization is one of the  department’s specific high-risk areas and is essential for addressing  many of the department’s other high-risk areas. For example, modernized  business systems are integral to the department’s efforts to address its  financial, supply chain, and information security management high-risk  areas.
subsections: section_title: Summary of NDAA Requirements
paragraphs: Congress included provisions in the NDAA, as amended, that are aimed  at ensuring DOD’s development of a well-defined BEA and associated  ETP, as well as the establishment and implementation of effective  investment management structures and processes. The act requires  DOD to, among other things, establish an investment approval and accountability structure along  with an investment review process, not obligate funds for a defense business system program with a total  cost in excess of $1 million unless the approval authority certifies that  the business system program meets specified conditions, develop a BEA that covers all defense business systems, develop an ETP for implementing the architecture, and identify systems information in DOD’s annual budget submissions.
The act also requires that the Secretary of Defense annually submit to the  congressional defense committees a report on the department’s  compliance with these provisions. DOD submitted its annual report to  Congress in March 2014, describing steps taken, under way, and planned  to address the act’s requirements.
section_title: DOD’s Approach to Business Systems Modernization
paragraphs: As of April 2014, the department’s approach to modernizing its business  systems environment, which is part of DOD’s overall effort to transform its  business operations, included improving business systems investment  management, reengineering the business processes supported by its  defense business systems, and improving and using the BEA and  associated ETP. These efforts are to be guided by DOD’s Chief  Management Officer and Deputy Chief Management Officer (DCMO).  Specifically, the Chief Management Officer’s responsibilities include  developing and maintaining a department-wide strategic plan for business  reform; establishing performance goals and measures for improving and  evaluating overall economy, efficiency, and effectiveness; and monitoring  and measuring the progress of the department. The DCMO’s  responsibilities include recommending to the Chief Management Officer  methodologies and measurement criteria to better synchronize, integrate,  and coordinate the business operations to ensure alignment in support of  the warfighting mission and developing and maintaining the department’s  enterprise architecture for its business mission area.
DOD has assigned roles and responsibilities to various governance  entities and positions related to business systems modernization. For  example, the Deputy’s Management Action Group is a senior-level forum  that meets several times a month to discuss department-wide  management issues, including business-related topics. This group is to  convene as the Defense Business Systems Management Committee  when it reviews defense business system portfolios. Under this  committee, the Defense Business Council (DBC) acts as a corporate- level investment review board (IRB) by overseeing the approach and  guidance for selecting and controlling the investment portfolio and making  recommendations on funds certifications. Table 1 describes selected  roles and responsibilities and composition of key governance entities and  positions related to business systems modernization.
In order to manage and oversee the department’s business operations  and approximately 1,180 covered defense business systems, the Office  of the DCMO developed its Integrated Business Framework. According  to the Office of the DCMO, this framework is used to align the  department’s strategic objectives—laid out in the National Security  Strategy, Quadrennial Defense Review, and Strategic Management  Plan—to its defense business system investments.
Using the overarching goals of the Strategic Management Plan, principle  staff assistants develop six functional strategies that cover eight  functional areas. These functional strategies define business outcomes,  priorities, measures, and standards for a given functional area within  DOD. The functional areas are
Acquisition,
Defense Security Enterprise,
Enterprise IT Infrastructure,
Financial Management,
Human Resources Management and Health Management,
Installations and Environment,
Logistics and Materiel Readiness, and
Security Cooperation.
The business objectives and compliance requirements laid out in each  functional strategy are to be integrated into the BEA.
The precertification authorities in the Air Force, Navy, Army, and other  departmental organizations use the functional strategies to guide the  development of organizational execution plans, which are to summarize  each component’s business strategy for each functional area. Each plan  includes a description of how the component’s goals and objectives align  with those in the functional strategies and the Strategic Management  Plan. In addition, each organizational execution plan includes a portfolio  of defense business system investments organized by functional area.  The components submit each of these portfolios of systems to the DBC  for certification on an annual basis.
According to DOD’s investment management guidance, the DBC reviews  the organizational execution plans and associated portfolios based on  four investment criteria—compliance, strategic alignment, utility, and  cost—to determine whether or not to recommend the portfolio for  certification of funding. The Vice Chairman of the Defense Business  Systems Management Committee approves certification decisions and  then documents those decisions in investment decision memoranda.  These memoranda state whether an individual organizational execution  plan has been certified, conditionally certified (i.e., obligation of funds has  been certified and approved but may be subject to conditions that restrict  the use of funds, a time line for obligation of funds, or mandatory changes  to the portfolio of business systems), or not certified (i.e., certification is  not approved due to misalignment with strategic direction, mission needs,  or other deficiencies).
subsections: section_title: The Office of the DCMO Currently Manages Business Systems, but a Planned Reorganization May Change These Responsibilities
paragraphs: The Office of the DCMO manages the day-to-day aspects of department- level business system oversight. It is currently composed of five  directorates with various responsibilities related to business systems  modernization. Table 2 shows the organizational components of the  Office of the DCMO and key responsibilities.
These responsibilities may shift as a result of planned organizational  changes. More specifically, on December 4, 2013, the Secretary of  Defense issued a memorandum that outlined the results of an  organizational review completed by the Office of the Secretary of  Defense. Among other things, the Secretary called for strengthening the  office of the DOD Chief Information Officer (CIO) by moving the oversight  of business systems from the DCMO to the CIO, and for strengthening  the Office of the DCMO to better coordinate the department’s business  affairs. According to the Director of Investment and Acquisition  Management in the Office of the DCMO, final decisions about how this  directive will be implemented have not yet been made. The memorandum  calls for all of the changes to occur by January 2015.
section_title: Overview of DOD’s BEA
paragraphs: DOD’s BEA is intended to serve as a blueprint for the department’s  business transformation efforts. In particular, the architecture is to guide  and constrain implementation of interoperable defense business systems  by, among other things, documenting the department’s business functions  and activities; the information needed to execute its functions and  activities; the business outcomes from using the BEA; and the list of  business rules, laws, regulations, and policies associated with its  business functions and activities. According to DOD, its architecture is  being developed using an incremental approach, where each new version  of the architecture addresses business mission area gaps or weaknesses  based on priorities identified by the department. The department’s BEA  focuses on documenting information associated with its end-to-end  business process areas (e.g., hire-to-retire and procure-to-pay).
The department considers its current approach to developing the BEA  both a “top down” and “bottom-up” approach. Specifically, according to  DOD, the architecture focuses on developing content to support  investment management and strategic decision making and oversight  (“top down”) while also responding to department needs associated with  supporting system implementation, system integration, and software  development (“bottom up”).
The department’s approach to developing its BEA involves the  development of a federated enterprise architecture, where member  architectures (e.g., Air Force, Army, and Navy) conform to an overarching  corporate or parent architecture and use a common vocabulary. This  approach is to provide governance across all business systems,  functions, and activities within the department and improve visibility  across DOD’s efforts.
section_title: Prior GAO Reviews of DOD’s Business Systems Modernization
paragraphs: Between 2005 and 2008, we reported that DOD had taken steps to  comply with key requirements of the NDAA relative to architecture  development, transition plan development, budgetary disclosure, and  investment review, and to satisfy relevant systems modernization  management guidance. However, each report also concluded that much  remained to be accomplished relative to the act’s requirements and  relevant guidance. We made recommendations to address each of the  areas, and DOD largely agreed with our recommendations.
However, in May 2009, we reported that the pace of DOD’s efforts in  defining and implementing key institutional modernization management  controls had slowed compared with progress made in each of the  previous 4 years, leaving much to be accomplished to fully implement the  act’s requirements and related guidance. In addition, between 2009 and  2012, we found that progress had been made, but long-standing  challenges we had previously identified remained to be addressed. For  example, the department’s budget submission for fiscal year 2013 did not  include all systems because of the lack of a reliable, comprehensive  inventory of all defense business systems. We concluded that DOD’s  progress in addressing the act’s requirements, its vision for a federated  architecture, and our related recommendations was limited, in part, by  continued uncertainty surrounding the department’s governance  mechanisms, such as roles and responsibilities of key organizations and  senior leadership positions. Accordingly, we made recommendations to  address the issues identified. DOD partially agreed with our  recommendations.
In 2013, we reported that DOD had continued to make improvements,  particularly by taking steps to establish a portfolio-based approach to  reviewing and certifying its defense business systems. However, among  other things, we also found that the BEA and ETP were still missing  important content and BEA and business process reengineering  certifications were not validated. We made additional recommendations in  these areas, and the department partially agreed with our  recommendations.
In addition, our most recent high-risk report noted that, while DOD’s  capability and performance relative to business systems modernization  had improved, significant challenges remained. For example, the  department had not fully defined and established a family of management  controls, such as corporate and component business architectures and  business system investment management processes. These  management controls are vital to ensuring that DOD can effectively and  efficiently manage an undertaking with the size, complexity, and  significance of its business systems modernization, and minimize the  associated risks. Furthermore, we also recently reported that, in order to  better identify and address potential duplication, DOD needed to develop  supporting component architectures, align them with its corporate  architecture to complete the federated BEA, and leverage its federated  architecture to avoid investments that provide similar, but duplicative,  functionality in support of common DOD activities.
section_title: DOD Has Addressed Most Provisions of the Act, but Needs to Continue Improving Its Investment Management Approach
paragraphs: DOD has made progress in addressing the provisions of the NDAA and  recommendations that we have made in recent years. Specifically, the  department issued its March 2014 Congressional Report on Defense  Business Operations, which describes updates and next steps for its  investment review process; improved the data used to manage its  business systems certification and approval process; issued an updated  ETP in December 2013; and issued its fiscal year 2015 funding request  for defense business systems. However, the department still faces  challenges in complying with key parts of the act and managing its  business systems. For example, the department needs to further refine its  approach for reviewing system certifications so that the reviews occur at  an appropriate level based on factors such as cost and complexity.  Without continued progress in improving its investment management  approach, DOD will be challenged in its ability to manage the billions of  dollars invested annually in modernizing its business system investments.
subsections: section_title: An Integrated Business Framework Guides DOD’s Business System Investments, but Further Refinements Are Needed to Improve the Investment Review Process
paragraphs: The act, as amended by the NDAA for Fiscal Year 2012, included  significant changes to the requirements for investment review and  certification of defense business systems. Specifically, it required DOD to  establish a department-wide IRB, chaired by the DCMO, and an  investment management process. The act also called for the use of  threshold criteria to ensure an appropriate level of review within the  department of, and accountability for, defense business system programs  depending on scope, complexity, and cost. In addition, IT investment  management best practices describe key practices for instituting an IRB,  including (1) grouping investments into portfolios, (2) coordinating the  investment management process with other internal management  controls, and (3) ensuring that systems receive an appropriate level of  review by developing tiered review boards.
Since we reported in May 2013, DOD has continued its efforts to  establish the IRB and further define and implement its defense business  system governance framework—called the Integrated Business  Framework—as discussed earlier in this report. This framework is used to  manage the department’s business operations and investments, and  includes six portfolios that align to functional areas. Of the three key  practices for establishing an IRB described above, DOD’s Integrated  Business Framework fully addresses one practice, partially addresses  one practice, and does not address one practice (see table 3).
Officials from the Office of the DCMO provided several reasons why two  of the key practices for instituting an IRB were not fully addressed. With  respect to aligning the Integrated Business Framework with the budget  process, officials stated that specific plans take time and coordination to  develop and implement. These officials noted that these efforts are  affected by the ongoing shift of responsibilities for business systems from  the Office of the DCMO to the DOD CIO. Notwithstanding the uncertainty  associated with this transition, until the department takes further steps to  align its business system investment review and budgeting processes, it  risks allocating resources to business system investments that might be  used more efficiently or effectively elsewhere across the department. In  addition, aligning the business system investment management process  with the budget process would improve the ability of components to guide  their investments from a strategic perspective and reduce pressure on the  DBC to approve the obligation of funds that have already been budgeted  for a given system.
Regarding the development of multiple review boards, officials from the  Office of the DCMO said that the DBC guidance does not call for  developing IRBs at multiple levels within the department because it is  important for the DBC to certify and approve all covered business  systems, and not just those that are more costly. These officials added  that some systems may be important for the business environment  regardless of their relatively small dollar value. In addition, these officials  stated that the Integrated Business Framework calls for reviewing  systems as part of system portfolios, not as individual systems. We agree  that the DBC should certify and approve all systems as called for by the  act, that it may want to occasionally focus greater attention on smaller  systems, and that the department should review systems as part of larger  system portfolios.
However, by not using a tiered review board approach, the department  may hamper the quality of the information submitted and the associated  reviews. This is evidenced by the results of the Office of the DCMO’s  validations of certifications discussed later in this report. Systems— including systems submitted as part of portfolios—that are larger in  scope, complexity, cost, or risk may benefit from additional scrutiny by the  DBC, while those that are less complex or lower cost could be reviewed  at the component level. Our IT investment management guidance notes  that enterprise-wide IT investment boards should be responsible for  systems that have high cost, high risk, or significant scope or duration,  while retaining responsibility and visibility into other system review  activities. However, lower-level boards could be chartered within business  units to oversee the investment management process for other systems  that are smaller or less costly, risky, or complex. While officials from the  Office of the DCMO stated that different systems are reviewed with  different levels of scrutiny based on various factors, those factors are not  documented in existing guidance. Until DOD ensures that investments  are reviewed at an appropriate level based on defined criteria, the DBC is  at an increased risk of failing to identify and address important issues  associated with large-scale and costly systems.
section_title: DOD Has Made Improvements in Data Used to Support Its Certification and Approval of Defense Business Systems, but Reviews Supporting These Decisions Can Be Improved
paragraphs: The NDAA requires that, prior to obligating funds, DOD certify that  investments meet certain conditions set out in the act. For fiscal year  2014, the department certified and approved most of its business systems  investments. In addition, it has made significant improvements to the data  used to manage information about these investments. However, while  DOD continued to assert the compliance of its investments with the  department’s BEA and business process reengineering, the quality of the  reviews supporting these assertions can be improved.
subsections: section_title: DOD Certified and Approved Most Business System Investments for Fiscal Year 2014
paragraphs: According to DOD’s March 2014 Congressional Report on Defense  Business Operations, the DBC reviewed certification requests totaling  $6.996 billion for 1,180 defense business systems across eight functional  areas. Of those requests, according to the March report, the council  approved $6.379 billion for 1,173 defense business systems and did not  approve $617 million associated with 40 business systems (see table 4).
These certifications included about $563 million in funds for fiscal year  2014 that the DBC conditionally certified (i.e., certified on the condition  that additional steps were taken). According to DOD’s March 2014 report,  the primary reason that systems were denied certification or conditionally  certified was that they were lacking a problem statement (or “business  need”). The following are examples of systems that were denied  certification or were conditionally certified in fiscal year 2014:
The Defense Logistics Agency’s Standard Procurement System  requested—but was denied—$3.6 million in development funds  because a problem statement had not been submitted. As of March  2014, no problem statement had been submitted, and the item  remained open.
The Defense Health Agency’s Armed Forces Health Longitudinal  Technology Application requested—and was conditionally approved  for—about $133.1 million for fiscal year 2014, with the condition that  the component submit a problem statement. As of April 2014, the  problem statement had been submitted and approved.
The Defense Logistics Agency’s Defense Retired and Annuitant Pay  System 2 requested a total of $12.6 million, but was only approved  for $2.2 million, with the condition that, among other things, the  component submit an updated cost estimate. As of March 2014, the  cost estimate had been submitted to the Office of the DCMO, but had  not yet been approved.
section_title: DOD Has Made Important Improvements to the Data Used to Manage Its Business System Investments
paragraphs: Our IT investment management framework states that to make good IT  investment decisions, an organization must be able to acquire pertinent  information about each investment and store that information in a  retrievable format, to be used in future investment decisions. As part of  this process, an organization should identify its IT assets and create a  comprehensive repository of investment information. The information in  the repository should identify each IT investment and its associated  components and should be accessible where it is of the most value to  those making decisions about IT investments.
DOD’s ability to address the act’s certification and approval requirements  depends in large part on its ability to capture and use data about business  systems and the department’s business system environment. To that end,  DOD maintains three key data sources or data repositories and requires  in its investment management guidance that components include updated  data within each source. Table 5 provides additional information about  each data source.
Since our May 2013 report, DOD has demonstrated important  improvements to the data it uses to manage its business systems. In May  2013, we reported that we were unable to verify the number of  certifications that occurred during the fiscal year 2013 certification and  approval process. At the time, DOD officials noted that certain data  fields in the original data sources were incomplete, incorrect, or  duplicative when certification requests were submitted. They also stated  that extensive manual cleanup efforts had been ongoing since the  beginning of the investment review process. Since then, the department  has established and used its DITIP system, which it developed in part as  a response to our recommendations associated with improving data  about business system investments. DITIP contains information on  certification decisions, including the amounts requested for certification  and subsequently certified (or not certified). According to officials from the  Office of the DCMO and DOD business system investment review  guidance, fiscal year 2014 certification requests and approval decisions  were all entered and maintained directly in this system. As a result,  information about investment certification requests and approval  decisions was consistently documented.
While this new approach has allowed DOD to improve certain information  about its business system certification and approvals, other information  needs continued improvement. Specifically, we have previously reported  that information about investment certifications and approvals was not  always reliable, and we have made recommendations aimed at improving  the department’s data about its business systems. For example,  according to officials from the Office of the DCMO, the numbers of  business systems reflected by DITIP and DITPR are not always  consistent. These officials stated that data issues persist in DITPR due to  the amount of data stored by the systems as well as the tight time frames  associated with implementing improvements. DOD intends to make  further updates to DITPR, including an update in the fourth quarter of  fiscal year 2014, which will include a data validation process intended to  align and reconcile the data common to DITPR and SNAP-IT via DITIP.
section_title: BEA and Business Process Reengineering Compliance Continues to Be Asserted for the Majority of Systems
paragraphs: Under the act, funds may not be obligated for covered business systems  unless the precertification authority asserts that, among other things, the  system is in compliance with the BEA and has undertaken appropriate  business process reengineering efforts. For fiscal year 2014 certification  reviews, investments were to follow the Office of the DCMO’s March 2013  BEA compliance guidance and September 2012 guidance on business  process reengineering. The BEA guidance describes an incremental  approach to BEA compliance requirements, including key elements  needed for asserting architecture compliance. The business process  reengineering guidance calls for specific levels of business process  reengineering for systems in different stages of development. For  example, prior to investing in development and modernization funds for a  defense business system, the components are required to submit a  problem statement to document and validate the system’s business  need. DOD’s guidance required submission of a form that includes  questions in areas related to various aspects of business process  reengineering, such as change management and target process  improvements.
For fiscal year 2014, DOD officials reported that 89 percent (1,053 of  1,180 systems) of defense business systems asserted compliance with  the BEA. Of the 44 systems that asserted non-compliance, 35 were  within one component of DOD—the Defense Finance and Accounting  Service. According to DOD officials, this component now has plans to  comply with BEA requirements and plans to update its assertions.
DOD’s business system investment review guidance calls for the  department to validate selected business system BEA assessments that  support the component’s assertions of compliance. We have previously  reported on the need to improve the quality of BEA compliance  assertions, and in 2013 we recommended that DOD implement and use  the BEA compliance assessments more effectively to support  organizational transformation efforts by, among other things, establishing  milestones by which selected validations of BEA compliance assertions  are to be completed. DOD partially agreed with this recommendation.
DOD selected 15 defense business systems for review, as called for in its  guidance. DCMO officials stated that of the 15 selected, all had asserted  that they complied with version 10.0 of the BEA. However, DCMO officials  found that 13 of the 15 selected had issues with their BEA compliance  assertions. Specifically, DCMO officials stated that they found that 9 of  the 13 had inconsistencies in the organizational execution plan or had  significantly incomplete compliance assertions in DOD’s BEA Compliance  System, and 4 of the 15 were assessed as having other opportunities for  improvements. According to DOD officials, some of these concerns were  addressed prior to system certification. They further stated that no  systems were denied funding certification due to incomplete BEA  compliance assertions because the Office of the DCMO worked with the  components to address the issues to the satisfaction of the DBC prior to  certification. Where additional action was needed, action items were  created and DCMO is tracking the progress of these systems’ efforts to  become compliant.
The Office of the DCMO’s validations of selected assessments are  important for helping to ensure that BEA assessments are accurate.  However, as demonstrated by these results, DOD needs to continue  working to ensure the quality of BEA assessments, as we have previously  recommended. Continued improvement in the BEA assessments will help  ensure that programs are being defined and implemented in a way that  facilitates interoperability and avoids duplication and overlap, which are  both goals of the BEA and the related investment management approach.
According to DOD officials, in fiscal year 2014, nearly 100 percent of  certified and approved defense business systems asserted that business  process reengineering had been completed. Further, the Office of the  DCMO also validated a sample of assessments associated with these  systems. According to DOD’s March 2014 report, this validation effort  showed that assertions were incomplete. Specifically, the DCMO selected  12 investments for validation, and the results of this effort were as follows:
Four systems received a “positive” rating, indicating that appropriate  business process reengineering was conducted.
Four systems received a “positive with recommendations” rating,  indicating that sufficient business process reengineering was  conducted but more could be done to improve the effort. DOD  recommended that the precertification authorities for these systems  ensure that appropriate business process reengineering compliance is  again assessed at the next milestone. For example, one system, the  Learning Management System, received a recommendation to  consider creating expected business outcomes, such as providing  career roadmaps to guide financial management employees in  planning and progressing in a DOD career. Further, another system,  the Next Generation Resource Management System did not provide  evidence of a change management plan, or documentation of the  resolution of the root causes identified in its “to-be” process models.
Four systems received a “could not validate” rating, indicating that the  business process reengineering assessment team was not able to  determine that a sufficient business process reengineering  assessment had been conducted. In these cases, DOD  recommended that the precertification authority ensure business  process reengineering assessment compliance is assessed at the  next acquisition milestone. For example, according to DOD officials,  the systems could not be validated because insufficient objective  evidence had been provided to substantiate their business process  reengineering assertions. According to the Office of the DCMO, three  of these four systems were in operations and maintenance, and it is  not often cost-effective to conduct business process reengineering  compliance reviews on investments in operations and maintenance.
The need for improvement demonstrated by some of DCMO’s selected  validations is consistent with our previous findings. Specifically, in May  2013 we reported on business process reengineering assertions of four  case study systems. All four assertions generally followed DOD’s  business process reengineering guidance; however, we also found that  limited documentation of root cause analyses was provided for all four.  We have previously made recommendations to improve DOD’s business  process reengineering-related efforts, including recommendations aimed  at the four systems we assessed in 2013.
The Office of the DCMO’s validations of selected systems’ business  process reengineering assessments are important for helping to ensure  that sufficient business process reengineering is conducted. However,  without further validation of the business process reengineering  assertions—as we have previously recommended—DOD is at risk of not  being able to accurately determine whether appropriate business process  reengineering has been undertaken, including ensuring that the business  processes supported by defense business systems have been  streamlined and have eliminated or reduced unique requirements to the  maximum practical extent.
section_title: DCMO Has Yet to Demonstrate That Its BEA Is Producing Business Value or Resulting in Elimination of Duplicative Systems
paragraphs: The act requires DOD to develop a BEA that covers all defense business  systems and their related functions and activities and includes key  information. For example, the act requires the BEA to include specific  information about data standards, policies and procedures, and  performance measures. In addition, the department seeks to use the BEA  to help guide, constrain, and enable interoperable business systems.  Moreover, we have previously emphasized that the BEA can be an  important tool for reducing potential overlap and duplication among DOD  business systems.
As we reported in May 2013, the current version of the BEA—Version  10.0— includes information aimed at meeting the act’s requirements,  such as data standards; business rules to help ensure compliance with  the laws, regulations, and policies incorporated in the BEA; and  performance measures associated with the department’s Strategic  Management Plan. However, as of April 2014, this version was still  missing other important content associated with achieving the  department’s goal of using the BEA to guide, constrain, and enable  interoperable business systems. For example, the BEA was missing  information about business systems associated with each of the  architecture’s business activities, and its business activities have not  been defined at a level that allows for more effective identification of  potential duplication and overlap. Adding such content is important for  various reasons, including improving the department’s ability to use the  architecture as a tool for managing its business systems investments by  more effectively identifying areas of potential duplication and overlap.
In addition, DOD has not yet demonstrated that the BEA has produced  business value for the department. According to the Director of  Investment Acquisition and Management and the Chief Architect,  previous versions of the BEA were designed to meet the requirements of  the act, but were not clearly focused on achieving business outcomes.  However, the act defines specific business outcomes that the BEA is  intended to support, such as integrating budget, accounting, and program  information and systems, and the department has defined additional BEA  goals, such as using the BEA to guide, constrain, and enable  interoperable business systems. These officials stated that in the future  they plan to use the BEA as a tool to better support interoperability, data  sharing, and reducing duplication. To help achieve this goal, according to  officials from the Office of the DCMO, DOD has established a process for  defining changes to the BEA that are intended to better support the  department’s ability to achieve business outcomes, and they first used  this process for defining updates to BEA 10.0. More specifically, the  Office of the DCMO established a BEA configuration control board— composed of representatives from the military departments and other  departmental offices—to make recommendations to the DBC on BEA  requirements and related content changes. Such a process may help the  department as it refines the architecture and seeks to achieve business  outcomes.
Further, while DOD has established a tool that can assist in identifying  potential duplication and overlap among business systems even with the  current gaps in BEA content, the department has not demonstrated that it  has used this information to reduce duplication and overlap. The April  2013 investment management guidance and a January 2013  memorandum issued by the DCMO established a single BEA compliance  tool—the BEA Compliance System—to document system-level BEA  compliance assertions and required all business systems submitted for  certification and approval for fiscal year 2014 to use this tool.
Officials from the Office of the DCMO stated that the results of BEA  compliance assessments conducted as part of the business system  certification and review process are available to staff across DOD to  review in making determinations about potential overlap and duplication.  For example, information provided by the department showed that, as of  February 2014, 120 systems performed activities associated with  maintaining asset information, 110 systems performed activities  associated with managing military health services, and 73 systems  performed activities associated with receiving and accepting a purchase  request. Officials from the Army and the Air Force stated that in fiscal  year 2014 they have begun to identify systems using BEA data that were  potentially duplicative. According to these officials, both the Army and the  Air Force have taken initial steps to phase out potentially duplicative  systems. However, the department has yet to demonstrate that it has  terminated these or other potentially duplicative programs that were  identified by using the BEA.
Officials from the Office of the DCMO offered several reasons why the  BEA compliance process has yet to result in the elimination of potentially  duplicative systems. For example, officials stated that it is difficult for the  DBC to make such determinations because information presented to the  council is high level and only points to potential instances of overlap and  duplication, which would require additional research. Instead, these  officials said, precertification authorities at the component level are in the  best position to determine whether their systems are duplicative of other  systems. However, these precertification authorities are not required to  identify and assess potential overlap and duplication before asserting  compliance with the BEA. In addition, the BEA continues to lack important  content, such as additional information about business activities, that  would further assist in identifying potential duplication and overlap among  business systems. Continued improvements to DOD’s BEA—as we have  previously recommended—and calling for more proactive identification  of potential overlap and duplication using information available in existing  tools can help the department use its BEA to identify and address  potential overlap and duplication among its billions of dollars in annual  defense business system investments.
section_title: Enterprise Transition Plan Content Has Been Improved and Is Consistent with Portfolio- Based Investment Management Approach
paragraphs: The act calls for the development of an ETP that implements the BEA and  covers all defense business systems and includes a listing of the  (1) new systems that are expected to be needed to complete the target  defense business systems computing environment, along with each  system’s performance measures, financial resource needs, and  risks or challenges to integration into the BEA;  (2) legacy systems that will be phased out of the defense business  systems computing environment within 3 years, together with the  schedule for terminating those legacy systems; and  (3) existing systems that are part of the target defense business systems  computing environment, as well as a strategy for making the  modifications to those systems that will be needed to ensure that such  systems comply with the defense BEA, including time-phased milestones,  performance measures, and financial resource needs.
The department’s fiscal year 2014 ETP, which was issued in December  2013 and uses data as of October 1, 2013, includes information about  1,179 covered defense business systems, including 79 legacy systems,  or those that will be terminated within 36 months. The ETP also includes  links to related documentation, including the functional strategies,  organizational execution plans, and investment decision memoranda.
DOD has taken steps to improve its ETP by including key content, but  has not included other information, as described below.
Milestones: The plan includes acquisition milestone information for  covered defense business systems (both legacy and core) through its  links to other systems, such as the Defense Acquisition Management  Information Retrieval system, DITIP, DITPR, and Office of  Management and Budget exhibit 300s. Specifically, according to  DOD’s April 2013 investment management guidance, system  owners are required to include start and end dates for life-cycle  phases of each system within DITPR.
Performance measures: The plan includes portfolio-level performance  measures in the organizational execution plans. For example, the Air  Force Human Resources Management organizational execution plan  includes performance measures associated with improving the  accuracy and timeliness of pay for airmen. However, these measures  are not linked to specific business systems.
Financial resource needs: The plan includes information about fiscal  year 2014 funding that was requested and ultimately approved under  the department’s business system investment review process for each  business system (both legacy and core).
Risks or challenges for BEA integration: The plan and related  documentation do not discuss each system’s risks or challenges to  integration into the BEA. Instead, the organizational execution plans  include portfolio-level risks and challenges, with only some related to  the BEA. For example, the Air Force Defense Security Enterprise  organizational execution plan identifies a challenge that the BEA and  business process reengineering process do not fit well with the  initiatives in this specific portfolio, and that many BEA and business  process reengineering compliance requirements are not applicable.
Termination dates: The ETP data also includes termination dates for  all 79 of the covered legacy systems and a listing of both new and  existing systems that will be part of the target defense business  systems computing environment.
Defining when and how the ETP will include the missing information—as  we have previously recommended—will help to ensure that DOD more  fully complies with the requirements of the act and help to inform  oversight of its business systems investments.
section_title: DOD’s Fiscal Year 2015 Budget Submission Includes Key Information on Business Systems
paragraphs: Another requirement of the NDAA for Fiscal Year 2005, as amended, is  that DOD’s annual IT budget submission must include key information on  each business system for which funding is being requested, such as the  system’s precertification authority and designated senior official, the  appropriation type and amount of funds associated with modernization  and current services (i.e., operation and maintenance), and the  associated Defense Business Systems Management Committee approval  decisions.
The department’s fiscal year 2015 budget submission includes a range of  information for business system investments requesting funding, such  as the system’s (1) name, (2) precertification authority, (3) designated  senior official, (4) approved funding for fiscal year 2014, and (5)  requested funding for fiscal year 2015. The submission also identifies the  amount of the fiscal year 2015 request that is for modernization versus  current services.
section_title: Conclusions
paragraphs: DOD has made important progress developing a framework to manage  the department’s business systems and more fully comply with the act,  yet additional and sustained efforts are necessary to fully achieve the  department’s modernization goals. Specifically:
The Office of the DCMO developed a portfolio-based investment  management framework—the Integrated Business Framework.  However, establishing specific plans for aligning the budgeting and  investment management processes would help ensure that systems  included in the budget also meet the department’s mission needs and  requirements, and may result in less money being budgeted for  systems that are duplicative or do not meet the needs of the  department. In addition, allowing less complex, costly, and risky  systems to be reviewed at the component level will give the DBC  more time to review more complex, costly, and risky systems.
The department also released a new system that should help to  address data reliability issues and certified most systems as meeting  the BEA and business process reengineering requirements.  Implementing our prior recommendations in this area will help DOD  improve the certification process.
The Office of the DCMO has taken steps to improve its ETP by  including most of the key content required by the act. Fully  implementing our prior recommendations will help to further inform  oversight of the department’s business systems investments.
The department formed a configuration control board for the BEA to  help manage future changes, which should provide decision makers  with better information on which to base their certification and  approval decisions. However, a more proactive use of the BEA—by  requiring precertification authorities to ensure that their systems do  not duplicate existing systems—will save time, money, and resources  by not investing in redundant systems.
Accordingly, further refinements to DOD’s business system investment  management process and more proactive use of the BEA to inform  decisions would assist the department in leveraging its limited resources  to more efficiently and effectively manage its business systems.
section_title: Recommendations for Executive Action
paragraphs: We are recommending that the Secretary of Defense direct the  appropriate DOD management entity to take the following three actions to  help ensure that the department’s business systems modernization  program is fully compliant with the act and is more effectively  implemented:
Define by when and how the department plans to align its business  system certification and approval process with its Planning,  Programming, Budgeting, and Execution process.
Define criteria for reviewing defense business systems at an  appropriate level in the department based on factors such as  complexity, scope, cost, and risk, in support of the certification and  approval process.
Develop guidance requiring military departments and other defense  organizations to use existing BEA content to more proactively identify  potential duplication and overlap.
section_title: Agency Comments and Our Evaluation
paragraphs: We received written comments on a draft of this report from DOD,  reprinted in appendix II. In its comments, the department concurred with  two recommendations and partially concurred with one recommendation.
DOD concurred with our first recommendation, to define by when and  how the department plans to align its business system certification and  approval process with its Planning, Programming, Budgeting, and  Execution process, and noted that further alignment of these processes  will continue in fiscal year 2015.
The department partially concurred with our second recommendation, to  define criteria for reviewing defense business systems at an appropriate  level in the department, in support of the certification and approval  process. In particular, DOD stated that it will continue to mature a process  that incorporates a variety of factors, including business process  complexity, risk, and portfolio costs. However, the department also stated  that our recommendation implies that it has not ensured that reviews  occur at an appropriate level. DOD noted that pre-certification authorities  are required to review the entire portfolio of investments before they are  presented to the DBC. The department added that these reviews enable  decision making, to include resourcing decisions at an organizational  level, which allows the DBC to focus on enterprise-level business issues  facing DOD. DOD also stated that, for fiscal year 2015, pre-certification  authorities will report the results of their review to the DBC, rather than  simply presenting their portfolio for certification.
We acknowledge that DOD’s guidance calls for reviews of investment  portfolios at multiple organizational levels, and our recommendation is not  intended to imply otherwise. Rather, its intent is to help ensure that DOD  establishes guidance and criteria for ensuring that detailed reviews of  individual systems occur at the appropriate levels within the department.  As described in this report, such an approach may involve establishing  investment review boards at multiple organizational levels. Such a tiered  review board approach would allow the DBC—acting at the executive- level IRB for the department—to focus its attention on more costly,  complex, or risky systems, while delegating detailed reviews of other  systems to lower organizational levels. By not using such an approach,  the department may not be able to ensure that the quality of the  information submitted for review is sufficient, and the associated  investment reviews may not be adequate for identifying and addressing  important issues associated with large-scale and costly systems. This is  evidenced by the results of the Office of the DCMO’s validations of BEA  and business process reengineering certifications discussed in this report.  While DOD’s planned actions reflect continued improvement in the  department’s oversight of defense business systems, these actions do  not fully address the recommendation. Accordingly, we believe that full  implementation of our recommendation is needed.
Finally, the department concurred with our recommendation to develop  guidance requiring military departments and other defense organizations  to use existing BEA content to more proactively identify potential  duplication and overlap. DOD added that the department plans to improve  its investment review process in fiscal year 2015 with a method that uses  BEA data to identify candidate systems for further review and, if  appropriate, retirement.
We are sending copies of this report to the appropriate congressional  committees; the Director, Office of Management and Budget; the  Secretary of Defense; and other interested parties. This report also is  available at no charge on the GAO website at http://www.gao.gov.
If you or your staff members have any questions on matters discussed in  this report, please contact me at (202) 512-4456 or chac@gao.gov.  Contact points for our Offices of Congressional Relations and Public  Affairs may be found on the last page of this report. GAO staff who made  major contributions to this report are listed in appendix III.
section_title: Appendix I: Objective, Scope, and Methodology
paragraphs: Our objective was to assess the actions taken by the Department of  Defense (DOD) to comply with section 332 of the National Defense  Authorization Act for Fiscal Year 2005 as amended. The specific  elements we assessed were (1) establishing a system investment  approval and accountability structure along with an investment review  process, (2) certifying and approving any business system program  costing in excess of $1 million, (3) developing a business enterprise  architecture (BEA) to cover all defense business systems, (4) developing  a transition plan for implementing the architecture, and (5) identifying  systems information in its annual budget submission.
To address the act’s provision for establishing an investment approval  and accountability review structure, along with an investment review  process, we reviewed the department’s current policies, procedures, and  guidance relative to the act’s requirements and criteria documented in our  information technology (IT) investment management framework.  Specifically, to determine whether the foundation for DOD’s new  investment management process had been fully established, we reviewed  and analyzed the department’s investment management guidance— published in April 2013—and the six functional strategies prepared for the  fiscal year 2014 certification process. We also reviewed DOD  documentation on future plans, including a draft version of its updated  investment management guidance and the March 2014 Congressional  Report on Defense Business Operations. We reviewed and analyzed  DOD’s investment management guidance to determine whether criteria  and procedures had been fully defined for making portfolio-based  investment decisions and the extent to which they aligned to our IT  investment management framework. Further we reviewed prior GAO  reports, DOD business system investment management guidance, and  DOD budget process policy to determine how the budget process aligned  with the business system investment review process. Finally, we  interviewed cognizant officials in the Office of the Deputy Chief  Management Officer (DCMO) to gain a better understanding of the  investment management process, its outputs, and planned improvements  for fiscal year 2015 defense business system certifications.
To determine whether the department was certifying and approving  business system programs costing in excess of $1 million in accordance  with the act’s provisions, we reviewed DOD’s March 2014 Congressional  Report on Defense Business Operations, and all functional strategies and  organizational execution plans, as well as their precertification request  memoranda, which were submitted as part of the fiscal year 2014  investment review process. We also reviewed investment decision  memoranda that documented the decisions made by the Defense  Business Council.
To determine whether DOD had made progress in improving the data  used to manage its business system investments, we reviewed data from  the department’s three authoritative data sources—the Defense  Information Technology Investment Portal (DITIP), the Defense  Information Technology Portfolio Repository (DITPR), and the Select and  Native Programming Data Input System for Information Technology  (SNAP-IT). We compared our results with the information reported in  DOD’s annual report to Congress to identify any discrepancies and  discussed these with cognizant officials from the Office of the DCMO.  Because DOD uses one system—DITIP—as the authoritative source for  information about defense business system certification and approvals,  we determined that the data were sufficiently reliable for our purposes.
As part of our evaluation under the certification requirement, we also  identified the processes associated with asserting BEA and business  process reengineering compliance for covered defense business systems  and the documentation used to support those assertions. To accomplish  this, we reviewed DOD guidance and related documentation that applied  to the fiscal year 2014 investment portfolio reviews and relevant updates  to that guidance issued during our audit. Further, we reviewed action  items, the status of those actions, selected business process  reengineering and BEA assertions, and data from DOD’s BEA  compliance assertion system— the BEA Compliance System— to  understand the extent to which defense business systems asserted  compliance with BEA requirements. In addition, we interviewed officials  from the Office of the DCMO about steps completed, under way, or  planned relative to BEA and business process reengineering compliance  and validation activities. We also reviewed DOD’s annual report to  Congress to determine if business process reengineering outcomes had  been reported.
To address the provision associated with developing the BEA, we  reviewed our previous findings associated with version 10.0 of the BEA,  released in February 2013, relative to the act’s architectural  requirements. We also identified recommendations from previous GAO  reports related to the act and measuring results and outcomes, and  gathered documentation to support the current status of those  recommendations. Specifically, we reviewed documentation related to  steps completed, under way, or planned to address the act’s  requirements and previously reported weaknesses, including slides  presented to the Defense Business Council regarding BEA requirements  criteria and updates, and the March 2014 Congressional Report on  Defense Business Operations. In addition, we reviewed documentation  related to efforts to achieve business value from the BEA, including  meeting minutes and charters from the BEA configuration control board  and sub-groups under that forum. To determine whether it is used to  identify and reduce potential overlap and duplication, we reviewed data in  DOD’s BEA Compliance System to determine how systems are mapping  to the department’s business functions, and reviewed DOD’s guidance on  BEA compliance. Finally, we interviewed officials from the Office of the  DCMO about planned updates and improvements to the BEA’s content,  and reasons that it has not yet been used to identify duplication and  overlap.
To address the provision associated with developing a transition plan, we  analyzed DOD’s enterprise transition plan, released in December 2013,  and associated guidance in DOD’s April 2013 investment management  guidance, relative to the act’s transition plan requirements and related  findings documented in previous GAO reports. Specifically, we reviewed  enterprise transition plan documentation, including functional strategies,  organizational execution plans, Office of Management and Budget exhibit  300s, and data from the DITIP and DITPR systems, to identify whether  that documentation met the requirements of the act. We also interviewed  officials from the Office of the DCMO to determine the steps completed,  under way, or planned to address the act’s requirements and previously  reported weaknesses.
To assess the system information in DOD’s fiscal year 2015 information  technology budget submission, we analyzed information contained in the  department’s fiscal year 2015 budget request, including the department’s  Section 332 Report and information contained in the systems that are  used to prepare its budget submission—SNAP-IT, DITPR, and DITIP— against the act’s requirements in this area. We also interviewed officials  from the Office of the DCMO and the Office of the Chief Information  Officer to discuss the accuracy and comprehensiveness of information  contained in the systems, and reasons for any discrepancies in the  information they contained.
We conducted this performance audit from October 2013 to May 2014, in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objective.
section_title: Appendix II: Comments from the Department of Defense
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact above, individuals making contributions to this  report include Michael Holland (Assistant Director), Camille Chaires,  Debra Conner, Kate Feild, Angel Ip, Anh Le, Lee McCracken, and  Christine San.