DOD Business Systems Modernization: Additional Action Needed to Achieve Intended Outcomes
Published: 2015-07-16T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: GAO designated DOD's multibillion dollar business systems modernization program as high risk in 1995, and since then has provided a series of recommendations aimed at strengthening its institutional approach to modernizing its business systems investments. Section 332 of the NDAA for fiscal year 2005, as amended, requires the department to take specific actions consistent with GAO's prior recommendations and included a provision for GAO to review DOD's efforts. In addition, the Senate Armed Services Committee Report for the NDAA for fiscal year 2015 included a provision for GAO to evaluate the usefulness and effectiveness of DOD's business enterprise architecture and business process reengineering processes. This report addresses both of those provisions.
In evaluating the department's compliance, GAO analyzed DOD's efforts to address open recommendations made in previous reviews. To evaluate the usefulness and effectiveness of the department's business enterprise architecture and business process reengineering processes, GAO surveyed the military department portfolio managers (24 in total) and interviewed officials. The response rate for the survey was 100 percent, making the results of the survey generalizable.
section_title: What GAO Found
paragraphs: The Department of Defense (DOD) has implemented 5 of the 16 recommendations made by GAO since June 2011 to address each of the overarching provisions for improving business systems management in the Ronald W. Reagan National Defense Authorization Act for Fiscal Year 2005 , as amended (NDAA) (10 U.S.C. § 2222) (see table). For example, it has implemented the recommendation to improve the data available for its business systems by making improvements to its repositories used for tracking information about the systems. Based on GAO's analysis, the department has partially implemented the remaining 11 recommendations. Implementing all recommended actions will improve DOD's modernization management controls and help fulfill the department's execution of the act's requirements.
Source: GAO analysis of DOD documentation. | GAO-15-627 .
DOD's business enterprise architecture and process reengineering efforts are not fully achieving the intended outcomes described in statute. More specifically, portfolio managers reported through GAO's survey that the architecture was not effective in constraining system investments or enabling DOD to produce reliable and timely information for decision-making purposes, among other things. As a result, the architecture has produced limited value. Portfolio managers reported that the department's business process reengineering efforts were moderately effective in streamlining business processes, but much less so in limiting the tailoring of commercial off-the-shelf systems. They also reported that these efforts have been useful in realizing selected benefits, such as improved documentation of business needs.
Managers GAO surveyed reported various challenges that impede the department's ability to fully achieve intended outcomes, such as cultural resistance to change and the lack of skilled staff. The department has work under way to address some of these challenges; however, gaps exist and the portfolio managers provided suggestions on how to close some of them. More fully addressing the challenges cited by the portfolio managers would help the department achieve better outcomes, including greater operational efficiencies and cost savings.
section_title: What GAO Recommends
paragraphs: GAO recommends that DOD utilize the results of the survey to determine additional actions that can improve management of its business process reengineering and enterprise architecture activities. DOD concurred with the recommendation.

section_title: Letter
paragraphs: For decades, the Department of Defense (DOD) has been challenged in  modernizing its business systems. Since the release of GAO’s 1995 High  Risk report, we have designated the department’s business systems  modernization program as high risk because of its vulnerability to fraud,  waste, abuse, and mismanagement, and because of missed opportunities  to achieve greater efficiencies. It continues to be on the high-risk list  today. In addition, we have reported numerous times that significant  potential exists for identifying and avoiding costs associated with  duplicative functionality across the department’s portfolio of defense  business systems environment. According to DOD, this environment  includes 2,179 business system investments and costs the department  approximately $10 billion in fiscal year 2015.
Since 1995, we have made a series of recommendations aimed at  strengthening DOD’s institutional approach to business systems  modernization and reducing the risks associated with key investments.  Further, Congress included provisions in the Ronald W. Reagan National  Defense Authorization Act for Fiscal Year 2005 (NDAA), as amended,  that were consistent with our past recommendations. Among other things,  section 332 of the act addresses DOD’s business system investment  review process, requirements for certifying and approving business  systems, and calls for the development of a business enterprise  architecture and associated enterprise transition plan. The act also directs  the Secretary of Defense to submit an annual report to congressional  defense committees on DOD’s compliance with certain requirements of  the NDAA for fiscal year 2005 not later than March 15 of each year,  through 2016, and included a provision for us to report on an assessment  of the department’s actions to comply with the requirements of the act. In  addition, the Senate Armed Services Committee report associated with  the fiscal year 2015 NDAA called for us to examine the utility and  effectiveness of DOD’s business enterprise architecture and business  process reengineering processes.
Accordingly, our objectives were to (1) assess the actions DOD has taken  to comply with section 332 of the NDAA for fiscal year 2005, as amended,  and (2) determine the usefulness and effectiveness of DOD’s business  enterprise architecture and business process reengineering processes.
To address the first objective, we reviewed DOD documentation and  interviewed department officials to examine DOD’s actions to implement  open recommendations from previous annual GAO reports that address  compliance with the act’s requirements.
For the second objective, we analyzed the fiscal year 2005 NDAA and  related DOD guidance to determine intended outcomes of the business  enterprise architecture and business process reengineering processes.  We then developed a structured data collection instrument (survey) based  on this analysis to obtain information from military department portfolio  managers on the extent to which DOD has achieved intended business  process reengineering and business enterprise architecture outcomes, as  well as information such as benefits achieved, challenges, and  improvement suggestions. We surveyed these military department  portfolio managers (24 in total) and received responses from all of them.  Accordingly, the survey results are generalizable. We then analyzed the  results to determine staff perspectives on the usefulness and  effectiveness of DOD’s processes. We also met with representatives from  selected DOD business system programs and cognizant DOD officials to  discuss their perspectives and relevant improvement efforts that are  planned and under way.
We conducted this performance audit from October 2014 to July 2015, in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objective. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objective. Additional details on our  objectives, scope, and methodology are contained in appendix I.
subsections: section_title: Background
paragraphs: DOD is one of the largest and most complex organizations in the world. In  support of its military operations, the department performs an assortment  of interrelated and interdependent business functions, including logistics  management, procurement, health care management, and financial  management. As we have previously reported, the DOD systems  environment that supports these business functions is complex and error  prone, and is characterized by (1) little standardization across the  department, (2) multiple systems performing the same tasks, (3) the  same data stored in multiple systems, and (4) the need for data to be  manually entered.
For fiscal year 2015, the department requested about $10 billion for its  business system investments. According to the department, as of April  2015, its environment includes approximately 2,179 business systems. Of  these systems, DOD reports that, for fiscal year 2015, the department  approved certification requests for 1,182 business systems covered by  the fiscal year 2005 NDAA’s certification and approval requirements.  Figure 1 shows how many of these 1,182 covered systems are  associated with each functional area.
DOD currently bears responsibility, in whole or in part, for about half (17  of 32) of the programs across the federal government that we have  designated as high risk. Seven of these areas are specific to the  department, and 10 other high-risk areas are shared with other federal  agencies.
Collectively, these high-risk areas in major business  operations are linked to the department’s ability to perform its overall  mission and affect the readiness and capabilities of U.S. military forces.  As such, DOD’s business systems modernization is one of the  department’s specific high-risk areas and is essential for addressing  many of the department’s other high-risk areas. For example, modernized  business systems are integral to the department’s efforts to address its  financial, supply chain, and information security management high-risk  areas.
subsections: section_title: The NDAA for Fiscal Year 2005 Requires DOD to Improve Its Management of Defense Business Systems
paragraphs: Congress included provisions in the fiscal year 2005 NDAA, as amended,  that are aimed at ensuring DOD’s development of a well-defined business  enterprise architecture and associated enterprise transition plan, as well  as the establishment and implementation of effective investment  management structures and processes. The act requires DOD to,  among other things, establish an investment approval and accountability structure along  with an investment review process; not obligate funds for a defense business system program with a total  cost in excess of $1 million over the period of the current future-years  defense program unless the approval authority certifies that the  business system program meets specified conditions, including  complying with the business enterprise architecture and having  appropriate business process reengineering conducted; develop a business enterprise architecture that covers all defense develop an enterprise transition plan for implementing the  architecture, and identify systems information in DOD’s annual budget  submissions.
The fiscal year 2005 NDAA also requires that the Secretary of Defense  submit an annual report to the congressional defense committees on the  department’s compliance with these provisions. DOD submitted its most  recent annual report to Congress on April 6, 2015, describing steps taken,  under way, and planned to address the act’s requirements.
section_title: DOD’s Approach to Business Systems Modernization
paragraphs: DOD’s approach to business systems modernization includes reviewing  systems annually to ensure that they comply with the fiscal year 2005  NDAA’s business enterprise architecture and business process  reengineering requirements. This effort includes both a certification of  compliance by lower-level department authorities and an approval of this  certification by higher-level department authorities. According to the act,  this certification and approval is to occur before systems are granted  permission to obligate funds for a given fiscal year. These efforts are to  be guided by DOD’s Chief Management Officer (CMO) and Deputy Chief  Management Officer (DCMO). Specifically, the CMO’s responsibilities  include developing and maintaining a departmentwide strategic plan for  business reform; establishing performance goals and measures for  improving and evaluating overall economy, efficiency, and effectiveness;  and monitoring and measuring the progress of the department.
The DCMO’s responsibilities include recommending to the CMO  methodologies and measurement criteria to better synchronize, integrate,  and coordinate the business operations to ensure alignment in support of  their warfighting mission and developing and maintaining the  department’s enterprise architecture for its business mission area. Table  1 describes selected roles and responsibilities and the composition of key  governance entities and positions related to business systems  modernization as they were documented for the fiscal year 2015 business  system certification and approval cycle.
Within the military departments, the entities described in table 1 are  supported by portfolio managers who oversee groups of business system  investments within specific functional areas. For example, the Department of the Navy’s financial management portfolio manager is  responsible for overseeing the Navy’s portfolio of financial management  systems.
section_title: Overview of DOD’s Integrated Business Framework
paragraphs: In order to manage and oversee the department’s business operations  and approximately 1,180 covered defense business systems, the Office  of the DCMO developed the Integrated Business Framework. According  to officials from the office, this framework is used to align the  department’s strategic objectives—laid out in the National Security  Strategy, Quadrennial Defense Review, and Strategic Management  Plan—with its defense business system investments.
Using the overarching goals of the Strategic Management Plan, principal  staff assistants developed six functional strategies that cover nine  functional areas. These functional strategies are to define business  outcomes, priorities, measures, and standards for a given functional area  within DOD. The business objectives and compliance requirements laid  out in each functional strategy are to be integrated into the business  enterprise architecture.
The precertification authorities in the Air Force, Navy, Army, and other  departmental organizations use the functional strategies to guide the  development of organizational execution plans, which are to summarize  each component’s business strategy for each functional area. Each plan  includes a description of how the component’s goals and objectives align  with those in the functional strategies and the Strategic Management  Plan. In addition, each organizational execution plan includes a portfolio  of defense business system investments organized by functional area.  The components submit each of these portfolios to the Defense Business  Council for certification on an annual basis.
According to the department’s 2015 Congressional Report on Defense  Business Operations, for the fiscal year 2015 certification and review  cycle, the department empowered the military department chief  management officers to manage their business systems portfolios and  conduct portfolio reviews. Results were presented to the Defense  Business Council and were to address topics such as major  improvements and cost reductions, return on investment, risks and  challenges, deviations from prior plans, and future goals.
According to DOD’s investment management guidance, for the fiscal year  2015 certification and approval cycle, the Defense Business Council was  to review the organizational execution plans and associated portfolios  based on four investment criteria—compliance, strategic alignment, utility,  and cost—to determine whether or not to recommend the portfolio for  certification of funding. The Vice Chairman of the Deputy’s Management  Action Group/Defense Business Systems Management Committee was to  approve certification decisions and then document the decision in an  investment decision memorandum. These memoranda were to indicate  whether an individual organizational execution plan has been certified;  conditionally certified (i.e., obligation of funds has been certified and  approved but may be subject to conditions that restrict the use of funds, a  time line for obligation of funds, or mandatory changes to the portfolio of  business systems); or not certified (i.e., certification is not approved due  to misalignment with strategic direction, mission needs, or other  deficiencies).
section_title: Overview of DOD’s Business Enterprise Architecture
paragraphs: DOD’s business enterprise architecture is intended to serve as a blueprint  for the department’s business transformation efforts. In particular, the  architecture is to guide and constrain implementation of interoperable  defense business systems by, among other things, documenting the  department’s business functions and activities and the business rules,  laws, regulations, and policies associated with them. According to DOD,  its architecture is being developed using an incremental approach, where  each new version of the architecture addresses business mission area  gaps or weaknesses based on priorities identified by the department. The  department’s business enterprise architecture focuses on documenting  information associated with its end-to-end business process areas (e.g.,  hire-to-retire and procure-to-pay). These end-to-end business process  areas may occur across the department’s nine functional areas. For  example, hire-to-retire occurs within the human resources management  functional area, while the cost management business process area  occurs across the acquisition, financial management, human resources  management, installations and environment, and logistics and materiel  readiness functional areas.
According to DOD officials, the current approach to developing the  business enterprise architecture is both a “top down” and “bottom-up”  approach. Specifically, the architecture focuses on developing content to  support investment management and strategic decision making and  oversight (top down) while also responding to department needs  associated with supporting system implementation, system integration,  and software development (bottom up). Consistent with DOD’s tiered  approach to business systems management, the department’s approach  to developing its business enterprise architecture involves the  development of a federated enterprise architecture, where member  architectures (e.g., Air Force, Army, and Navy) conform to an overarching  corporate or parent architecture and use a common vocabulary. This  approach is to provide governance across all business systems,  functions, and activities within the department and improve visibility  across the respective efforts.
section_title: Overview of DOD’s Business Process Reengineering Efforts
paragraphs: DOD defines business process reengineering as a logical methodology  for assessing process weaknesses, identifying gaps, and implementing  opportunities to streamline and improve the processes to create a solid  foundation for success in changes to the full spectrum of operations.  DOD’s reengineering efforts are intended to help the department  rationalize its covered business system portfolio, improve its use of  performance management, control scope changes, and reduce the cost  of fielding business capability. According to DOD officials, the department  has taken a holistic approach to business process reengineering, which  includes a portfolio and end-to-end perspective. It has also issued  business process reengineering guidance that calls for alignment of  defense business systems within the Organizational Execution Plan to its  functional strategy’s strategic goals.
An important component of the department’s business process  reengineering efforts is the problem statement development and review  process. A problem statement is developed when a defense business  system is seeking certification for a development or modernization effort.  The statement is to include, among other things, a description of the  problem that the system intends to address and a discussion of the costs,  benefits, and risks of various alternatives that were considered. As part of  the annual certification and approval process, problem statements are to  be reviewed to support that appropriate business process reengineering  has been conducted on investments seeking certification.
section_title: DOD Has Made Progress Strengthening Its Management of Business Systems, but Needs to Continue Implementing Recommended Improvements
paragraphs: The department has implemented 5 of the 16 recommendations that GAO  has made since June 2011 to address each of the overarching provisions  for improving business systems management in the fiscal year 2005  NDAA. The fiscal year 2005 NDAA, as amended, includes provisions  associated with developing a business enterprise architecture and  enterprise transition plan, improving the department’s investment  management structures and processes, improving its efforts to certify  defense business systems, and mandated budgetary reporting. Since  2011, we have issued four reports in response to the act’s requirement  that we assess the actions taken by the department to comply with the   In those reports, we have made recommendations to  act’s provisions.address each of the act’s overarching provisions for improving business  systems management. Table 2 identifies the recommendations we have  made since 2011 associated with the fiscal year 2005 NDAA.
Table 3 presents a summary of the current status of these  recommendations. Appendix II provides additional information about the  status of each recommendation.
As of April 2015, the department had implemented 5 of the 16  recommendations that we have made since June 2011. For example, the  department has implemented the recommendation to improve its  reporting of business system data in its annual budget request. In  particular, the department has established common elements in its three  primary repositories used for tracking information about business  systems, which allows information about individual business systems to  be matched across systems. In addition, the Office of the CIO  demonstrated that it conducts periodic data quality assessments. As a  result, the department is better positioned to report more reliable  information in its annual budget request and to maintain more accurate  information about business systems to support its efforts to manage  them.
In addition, the department has improved the alignment of its Planning,  Programming, Budgeting, and Execution process with its business  systems certification and approval process. For example, according to the  department’s February 2015 certification and approval guidance,  Organization Execution Plans are to include information about  certification requests for the upcoming fiscal year as well as over the  course of the Future Years Defense Program. As a result, the  department’s business system certification and approval process can  support better informed decisions about system certifications and inform  recommendations on the resources provided to defense business  systems as part of the Planning, Programming, Budgeting, and Execution  process.
The department has partially implemented the remaining 11  recommendations. For example, the department’s February 2015  investment management guidance, which describes DOD’s business  system certification and approval process, identifies four criteria and  specifies the associated assessments that are to be conducted when  reviewing and evaluating component-level organizational execution plans  in order to make a portfolio-based investment decision. The guidance  also states that return on investment should be considered when  evaluating program cost. However, it does not call for the use of actual- versus-expected performance data and predetermined thresholds.
Further, the Office of the DCMO has developed a draft resource  allocation plan for each of its directorates and their respective divisions.  This draft plan includes staffing profiles that describe each division’s  needed staff competencies and qualifications. However, the Office of the  DCMO did not demonstrate that it has addressed other important aspects  of strategic human capital planning. For example, the office did not  demonstrate that it has developed a skills inventory, needs assessment,  gap analysis, and plan to address identified gaps, as called for by our  recommendation. Appendix II provides additional information about the  recommendations that DOD has fully and partially implemented.
Implementing the remaining 11 recommendations will improve DOD’s  modernization management controls and help fulfill the department’s  execution of the requirements of the act.
section_title: DOD Needs to Improve the Usefulness and Effectiveness of Its Business Enterprise Architecture and Business Process Reengineering Initiatives
paragraphs: DOD’s business enterprise architecture and process reengineering efforts  are not fully achieving the intended outcomes described in statute. More  specifically, with respect to the architecture, portfolio managers  (managers) we surveyed reported that it was generally not effective in  achieving its intended outcomes and that its usefulness in achieving  benefits, such as reducing the number of applications, was limited. With  respect to process reengineering, managers reported these efforts were  moderately effective at streamlining business processes, but less so in  limiting the need to tailor commercial off-the-shelf systems.
Portfolio managers cited a number of challenges impeding the usefulness  and effectiveness of these two initiatives, such as the availability of  training, lack of skilled staff, parochialism, and cultural resistance to  change. DOD has various improvement efforts under way to address  some of these challenges; however, additional work is needed and the  managers provided some suggestions for closing the gap. More fully  addressing the cited challenges would help increase the utility and  effectiveness of these initiatives in driving greater operational efficiencies  and savings. Appendix I provides additional details about our survey  methodology.
subsections: section_title: The NDAA for Fiscal Year 2005 Defines Intended Outcomes for the Business Enterprise Architecture
paragraphs: The fiscal year 2005 NDAA, as amended, requires DOD to develop a  business enterprise architecture that covers all defense business systems  and will be used as a guide for these systems. According to the act, the  architecture is intended to help achieve the following outcomes:
Enable DOD to comply with all applicable laws, including federal  accounting, financial management, and reporting requirements.
Guide, permit, and constrain the implementation of interoperable  defense business systems.
Enable DOD to routinely produce timely, accurate, and reliable  business and financial information for management purposes.
Facilitate the integration of budget, accounting, and program  information and systems.
Provide for the systematic measurement of performance, including the  ability to produce timely, relevant, and reliable cost information.
The act also specifies that the department is not to obligate funds for  defense business system programs that have a total cost in excess of $1  million unless the system’s approval authority certifies that the program  complies with the business enterprise architecture and the certification is  subsequently approved by the department’s Investment Review Board.
Achieving the act’s intended outcomes would contribute to the  department’s ability to use the architecture to realize important benefits  that we and others have previously identified, such as cost savings or   For example, if the architecture effectively guides, permits,  avoidance.and constrains the implementation of interoperable systems, that would  contribute to increased information sharing and improved system  interoperability. As another example, using the architecture to produce  timely and reliable business and financial information would contribute to  improving management decisions associated with enhanced productivity  and improved business and IT alignment, among other things.
section_title: The Business Enterprise Architecture Has Produced Limited Value, According to Portfolio Managers
paragraphs: The majority of DOD portfolio managers we surveyed reported that the  business enterprise architecture has not been effective in meeting its  intended outcomes. More specifically, half of the managers surveyed  reported that the business enterprise architecture was effective in  enabling compliance with all applicable laws. However, fewer than 40  percent reported that the architecture was effective in helping to achieve  the other outcomes called for by the fiscal year 2005 NDAA. Table 4  provides additional information on survey responses regarding the act’s  specific requirements.
Portfolio managers provided additional details to further explain their  survey responses. Their comments included the following:
The architecture is a standalone effort that does not drive  comprehensive portfolio and business management through the  various DOD components.
The architecture is overwhelming to review and is not integrated with  other activities that occur throughout the remainder of the year.
The compliance requirements are not sufficiently defined to enable  system interoperability.
Portfolio managers also reported that the usefulness of DOD’s business  enterprise architecture in achieving various potential benefits is limited.  For example, 75 percent reported limited achievement of improved  change management and 74 percent reported limited achievement of  streamlined end-to-end business processes. In addition, 71 percent  reported limited achievement of benefits such as a reduced number of  applications, improved business and IT alignment, enhanced productivity,  and achieving financial benefits such as cost savings or cost avoidance.  Table 5 summarizes the portfolio managers’ survey responses.
Although managers reported limited achievement of benefits, two  provided specific examples of individual benefits associated with the  business enterprise architecture. More specifically, one cited saving $10  million annually due to the establishment of a DOD-wide military housing  system that has replaced a number of individual systems. A second  reported $11.5 million in architecture-related savings through the  retirement of 48 real property and financial management systems.
In addition, officials from the Office of the DCMO provided specific  examples of benefits that they stated can be attributed, at least in part, to  the department’s business architecture. For example, according to these  officials, two proposed new defense business system investments were  not approved by DOD due, in part, to architecture reviews that revealed  the requested capabilities were already available in existing systems.
subsections: section_title: Numerous Challenges in Achieving Intended Business Enterprise Architecture Outcomes Identified by Portfolio Managers
paragraphs: The surveyed DOD portfolio managers reported that their functional areas  face many challenges in achieving the outcomes described in the NDAA  for fiscal year 2005. The most frequently cited challenges reported were  the usability of the compliance tool (79 percent), frequent changes to the  architecture (75 percent), the availability of training (71 percent), the  availability of skilled staff (71 percent), parochialism (67 percent), and  cultural resistance to change (63 percent). Table 6 identifies the survey  responses to achieving the architecture’s intended outcomes.
Officials from the Office of the DCMO, including the Lead Architect for the  business enterprise architecture and the Chief of Portfolio Management,  described various efforts under way to address selected challenges  identified in our survey results. With regard to the top ranked challenge  (usability of DOD’s architecture compliance tool), the office has been  working on a more robust replacement tool. As of April 2015, the office  had moved architecture content and associated compliance information  from its previous tool into its Integrated Business Framework-Data  Alignment Portal. Further, the department plans to require all fiscal year  2016 compliance assessments to be completed in this portal  environment.
According to officials from the Office of the DCMO, this change will help  ensure that architecture-related information is available in the same  place, which will help support more sophisticated analysis of information  about business systems. For example, by combining information about  the architecture, compliance information, functional strategies, and  organizational execution plans, the department could more easily conduct  analyses that will help support portfolio management. According to these  officials, examples of such analyses include the ability to identify the  funds certified and approved for various business activities and the ability  to identify systems that conduct similar system functions.
With regard to the challenge associated with limited alignment between  corporate and component architectures, the officials from the Office of the  DCMO stated that they intend to develop an overarching (or federated)  architecture that will capture content from, and allow governance across,  the department (e.g., Army, Navy, and Air Force). We previously  recommended that DOD establish a plan for how it will address business  enterprise architecture federation in 2013.
The department’s improvement efforts only address selected reported  challenges. However, portfolio managers offered a number of  suggestions that relate to other identified challenges that may help close  gaps in these efforts. Key suggestions included:  Improve tools: Four of 24 managers offered suggestions that relate  to compliance tool usability. For example, one portfolio manager  stated that functionality should be added to the architecture  compliance tool to automatically create and build the architecture  artifacts mentioned in compliance guidance using the information  already included in the tool for each system. Another portfolio  manager stated that there are no tools available that portfolio  managers can use to analyze their portfolios relative to the  architecture.
Provide additional training: Two managers offered suggestions  associated with additional training. For example, one manager  reported that the compliance tool is not user friendly and little to no  training was offered when programs were required to use it to assert  compliance. As a result, this manager added that more training should  be made available for using the compliance tool.
Start the process earlier in a system’s life cycle: One manager  suggested the architecture be addressed earlier in the acquisition life  cycle, such as in the analysis of alternatives phase, in order to help  assess whether existing solutions are already employed in other  areas of the enterprise. If the architecture compliance process  uncovers potential duplication or overlap, it might be easier to stop  development of a duplicative system earlier in its life cycle rather than  waiting until a business process is more reliant on a planned system  that is closer to becoming operational.
Establish priorities: One portfolio manager suggested that the  department develop departmental business improvement and  integration priorities and develop clearly understandable and  verifiable compliance standards that will guide and constrain systems  development to help achieve those priorities.
Improve guidance: Two managers suggested that the department  improve its guidance to clarify the documentation that systems  developed prior to the existence of the business enterprise  architecture are required to prepare to address the business  enterprise architecture compliance requirement.
Improve content: Seven managers offered suggestions associated  with improving content. For example, one manager stated that the  business enterprise architecture is large and cumbersome and  incomplete in many areas.
Addressing the challenges cited by the portfolio managers could help  increase the utility and effectiveness of the department’s business  enterprise architecture in driving greater operational efficiencies and cost  savings.
section_title: The NDAA for Fiscal Year 2005 Defines Intended Outcomes for Business Process Reengineering
paragraphs: The fiscal year 2005 NDAA, as amended, establishes expected outcomes  for the department’s business process reengineering efforts. The act  states that funds for covered business system programs cannot be  certified and approved unless each program’s pre-certification authority  has determined that, among other things, appropriate business process  reengineering efforts have been undertaken to ensure that the business  process supported by the program is, or will be, as streamlined and  efficient as practicable and the need to tailor commercial off-the-shelf  systems to (a) meet unique requirements, (b) incorporate unique  requirements, or (c) incorporate unique interfaces has been eliminated or  reduced to the maximum extent practicable. As we have previously  reported, modifications to commercial off-the-shelf systems should be  avoided to the extent practicable as they can be costly to implement.
Achieving the intended outcomes of the fiscal year 2005 NDAA would  increase the department’s ability to realize key benefits to business  systems modernization. For example, reengineering business processes  to be as streamlined as possible can result in increased efficiencies, a  reduced number of interfaces, and decreased program costs.
section_title: DOD’s Reengineering Efforts Have Moderately Helped to Streamline Business Processes and Realize Some Efficiencies, but More Work Is Needed to Fully Achieve Intended Outcomes, According to Portfolio Managers
paragraphs: The department’s business process reengineering efforts have had mixed  success in achieving their intended outcomes. Specifically, 63 percent of  the portfolio managers we surveyed reported that the efforts were  effective in helping to ensure that the business processes supported by  the defense business systems they manage are (or will be) streamlined  and efficient as practicable. As an example, one manager reported this  effort highlighted the strengths and weaknesses of the systems within  their specific portfolio. Another reported that their portfolio has been  reduced from 147 systems to 13 due, in part, to the business process  reengineering efforts.
However, the general consensus among surveyed portfolio managers  was that the department’s efforts were less effective in helping to limit  tailoring of commercial off-the-shelf systems. Only 29 percent reported  that DOD’s business process reengineering efforts were effective in  eliminating or reducing the need to tailor commercial off-the-shelf  systems. Tailoring might be required, for example, because existing  policy and guidance might limit a system’s ability to conform to a specific  approach for executing a business process that is already built into an  individual commercial off-the-shelf system. Another reason given was that  managers have limited knowledge about the commercial off-the-shelf  products that are available via established enterprise licenses and this  limited knowledge makes it difficult to conduct effective business process  reengineering.
Table 7 provides additional information on portfolio managers’ responses  regarding the effectiveness of DOD’s business process reengineering  efforts.
Portfolio managers reported that business process reengineering has  been useful in helping to achieve selected benefits. In particular, 70  percent reported that efforts have resulted in streamlined business  processes. Sixty-seven percent reported that efforts have resulted in  improved documentation of business needs, which is consistent with  DOD’s focus on developing problem statements for new capabilities.  Such problem statements reflect analysis of a perceived business  problem, capability gap, or opportunity. According to officials from the  Office of the DCMO, they help ensure that programs are aligned with  DOD’s strategic needs, and also assist the department’s efforts in  identifying redundancies and duplication. However, only 29 percent of the  portfolio managers surveyed reported that efforts to reduce program costs  have been effective. Table 8 summarizes the portfolio managers’ survey  responses.
The surveyed DOD portfolio managers identified a range of challenges to  fully achieving the business process reengineering outcomes described in  the fiscal year 2005 NDAA. In particular, cultural resistance to change  was the most frequently cited challenge (71 percent), followed by  parochialism (i.e., focusing on one’s own sub-organization rather than  having an enterprise-wide view.), availability of skilled staff, and  availability of training (all at 67 percent). The quality of business process  reengineering compliance guidance, the compliance review process, and  the timing of the reengineering relative to system development work were  also reported as important challenges (all at 63 percent).
Table 9 summarizes survey responses to questions about the challenges  to business process reengineering.
DOD has taken steps to improve its reengineering efforts that may, in  part, address some of the challenges identified in our survey results. With  regard to parochialism (i.e., focusing on one’s own sub-organization  rather than having an enterprisewide view), the department is developing  online tools that provide additional information to program managers,  portfolio managers, pre-certification authorities, and the Defense  Business Council. For example, the department’s problem statement  portal is to be a repository for problem statement submissions and is to  be available departmentwide. In addition, the department has developed  its Integrated Business Framework-Data Alignment Portal, which is to  provide, among other things, additional information about individual  business systems, such as information about which systems execute  specific business activities and system functions.
Further, with respect to addressing the challenge associated with the  business process reengineering compliance review process, the  department has taken steps to help ensure improved accountability for a  portion of certification and approval requests. In particular, according to  officials from the Office of the DCMO, the DCMO allowed the military  departments more autonomy and responsibility for reviewing their system  portfolios during fiscal year 2015 certification and approval reviews.  Nevertheless, as we have previously reported, and as discussed in  appendix II, this process is not guided by specific criteria for elevating  certain systems to the Defense Business Council that might require  additional oversight.
Notwithstanding these improvement efforts, as reported in feedback by  the military department portfolio managers, additional work is needed.  These managers provided a number of suggestions to help address the  identified challenges. Suggestions included:  Improve business process reengineering training: Two portfolio  managers offered suggestions that relate to improved training. For  example, one manager stated that the department should establish  minimum training standards.
Improve business process reengineering guidance: Two  managers offered suggestions associated with improved guidance.  For example, one portfolio manager stated that sufficient guidance  does not exist to describe meaningful business process models or  how such models should be analyzed.
Align business process reengineering with system development  activities: One portfolio manager stated that the reengineering  process should be more closely tied to acquisition milestones instead  of being assessed on an annual basis.
According to GAO’s standards for internal controls,ensure that there are adequate means of obtaining information from  stakeholders that may have a significant impact on the agency achieving  its goals. While we did not evaluate the effectiveness of these  suggestions, they may be valuable for the Office of the DCMO to consider  in its ongoing and future business process reengineering improvement  efforts. More fully addressing the challenges cited by the portfolio  managers would help the department achieve better outcomes, including  limiting the tailoring of commercial off-the-shelf systems.
section_title: Conclusions
paragraphs: DOD has made progress in improving its compliance with section 332 of  the NDAA for fiscal year 2005, as amended. Specifically, the department  has implemented 5 of the 16 recommendations that we have made since  2011 that are consistent with the requirements of the act and has partially  implemented the remaining 11 recommendations. The recommendations  not fully implemented relate to improving the department’s investment  management processes and efforts to certify defense business systems,  among other things. Fully implementing them will help improve DOD’s  modernization management controls and fulfill the department’s execution  of the act’s requirements.
Collectively, DOD’s business enterprise architecture and business  process reengineering efforts show mixed results in their effectiveness  and usefulness in achieving the intended outcomes and benefits. Among  other things, portfolio managers reported that the architecture does not  enable DOD to produce reliable and timely information for decision- making purposes. Additionally, DOD’s reengineering efforts are effective  in streamlining business processes, but not in reducing the tailoring of  commercial software products. Portfolio managers reported that various  challenges exist in achieving intended outcomes and benefits, including  cultural resistance, parochialism, and a lack of skilled staff. DOD has  various improvement efforts under way; however, gaps exist and portfolio  managers provided suggestions on how to close some of them. Until  these gaps are addressed, the department’s ability to achieve important  outcomes and benefits will continue to be limited.
section_title: Recommendation for Executive Action
paragraphs: To help ensure that the department can better achieve business process  reengineering and enterprise architecture outcomes and benefits, we  recommend that the Secretary of Defense utilize the results of our  portfolio manager survey to determine additional actions that can improve  the department’s management of its business process reengineering and  enterprise architecture activities.
section_title: Agency Comments and Our Evaluation
paragraphs: We received written comments on a draft of this report from DOD’s  Deputy Chief Management Officer (DCMO). The comments are reprinted  in appendix III.
In the comments, the DCMO concurred with our recommendation and  stated that the department will use the results of our portfolio manager  survey to help make improvements. The DCMO also described  associated improvement efforts. For example, the DCMO stated that the  department plans to restructure the Business Enterprise Architecture to  focus more explicitly on the business processes being executed within the  functional domains, which span all levels of the department. DOD officials  also provided technical comments, which we have incorporated as  appropriate.
We are sending copies of this report to the appropriate congressional  committees; the Director, Office of Management and Budget; the  Secretary of Defense; and other interested parties. This report also is  available at no charge on the GAO website at http://www.gao.gov. If you  or your staff members have any questions on matters discussed in this  report, please contact me at (202) 512-4456 or chac@gao.gov. Contact  points for our Offices of Congressional Relations and Public Affairs may  be found on the last page of this report. GAO staff who made major  contributions to this report are listed in appendix IV.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to (1) assess the actions by the Department of  Defense to comply with section 332 of the National Defense Authorization  Act (NDAA) for Fiscal Year 2005, as amended and (2) determine the  usefulness and effectiveness of DOD’s business enterprise architecture  and business process reengineering processes.
To address the first objective, we identified recommendations related to  DOD’s business systems modernization efforts that we made in our  annual reports from 2011 to 2014 (16 recommendations total) in response  to the fiscal year 2005 NDAA’s requirements. Though we have made  recommendations in this area prior to 2011, those recommendations have  since been closed. We evaluated the department’s written responses and  related documentation on steps completed to implement or partially  implement the recommendations. Documentation we analyzed included  guidance on business enterprise architecture and business process  reengineering compliance, guidance on certifying and approving defense  business systems, and documentation about the department’s problem  statement development and review process. In addition, we interviewed  officials from the Office of the Deputy Chief Management Officer and the  Office of the Chief Information Officer, and observed a demonstration of  the Office of the Deputy Chief Management Officer’s Integrated Business  Framework-Data Alignment Portal tool to better understand the actions  taken to address our recommendations. We also reviewed the  department’s annual report to Congress, which was submitted on April 6,  2015, to identify gaps or inconsistencies with the implementation of the 16  recommendations.
To address our second objective, we determined the intended outcomes  of the business enterprise architecture and business process  reengineering processes by analyzing the fiscal year 2005 NDAA. We  also determined potential benefits associated with the processes by  reviewing department guidance on the processes and related  documentation. This includes DOD’s business enterprise architecture and  business process reengineering guidance, Defense Business System  Investment Management Process Guidance, the Business Case Analysis  Template, DOD’s Business Enterprise Architecture 10.0 AV-1 Overview  and Summary Information, the department’s Strategic Management Plan,  and the Information Resource Management Strategic Plan for fiscal years  2014 and 2015. We also reviewed relevant GAO reports on business  enterprise architecture and business process reengineering.
We then developed a structured data collection instrument (survey) to  gather information on the usefulness of the two specified IT  modernization management controls at DOD in achieving their intended  outcomes and their effectiveness in achieving associated benefits. As  part of this survey, we also developed questions to help us determine (1)  challenges related to complying with the processes and (2) suggestions  for achieving business enterprise architecture and business process  reengineering outcomes, including suggestions for achieving these  outcomes in a more cost-effective manner.
Selected questions contained a ratings scale for managers to choose a  response that was consistent with the aforementioned topic areas. For  example, we asked managers to rate the effectiveness of the business  enterprise architecture and business process reengineering efforts using  a scale containing the following choices: neither effective nor ineffective, not applicable/no basis to judge.
We also asked managers to identify the extent to which their portfolios  had achieved benefits associated with business enterprise architecture  and business process reengineering efforts using a scale containing the  following choices:  little or no extent, or not applicable/no basis to judge.
We pre-tested the questions with various DOD officials including officials  from the Office of the Deputy Chief Management Officer, and with  portfolio and program-level officials within the military departments. As a  result, we determined that the military department portfolio managers  were in the best position to answer our questions because they manage  and have a perspective across an entire portfolio of defense business  systems. Officials from DCMO’s Management, Policy, and Analysis  Directorate provided us with a list of portfolio managers for the three  military departments. We did not include portfolio managers for DOD  entities outside of the military departments.
We obtained responses from all surveyed portfolio managers (24 in  total). Accordingly, these results are generalizable. We analyzed and  summarized the survey results to help determine the usefulness and  effectiveness of DOD’s business process reengineering and enterprise  architecture efforts, as well as related challenges and suggestions for  improvement. In addition, though we collected examples of cost savings  estimates from managers, and cite them in the report, we did not assess  the cited cost savings estimates.
We also met with managers of selected DOD business system programs  and other knowledgeable DOD officials to discuss their perspectives on  DOD’s business enterprise architecture and business process  reengineering efforts. This included interviewing officials associated with  defense business programs from each of the military departments and  from across various business functions, including program managers,  enterprise architects, and other technical and program operations  officials. Further, when available, we reviewed documentation provided by  DOD program managers to substantiate answers provided as part of our  interviews. We also discussed the survey results with officials from the  Office of the DCMO to obtain their perspectives on the results and  discussed with these officials ongoing efforts to improve the department’s  business process reengineering and enterprise architecture efforts.
The practical difficulties of conducting any survey may introduce errors,  commonly referred to as nonsampling errors. For example, difficulties in  how a particular question is interpreted, in the sources of information that  are available to respondents, or in how the survey data are analyzed can  all introduce unwanted variability into survey results. To minimize such  nonsampling errors, a social science survey specialist designed the  questionnaire in collaboration with GAO staff with subject matter  expertise. As stated earlier, the questionnaire was pre-tested to ensure  that the questions were relevant, clearly stated, and easy to comprehend.  When data from the survey were analyzed, an independent analyst  reviewed the computer program used for the analysis of the survey data.  Since this was a web-based survey, respondents entered their answers  directly into the electronic questionnaire, thereby eliminating the need to  have the data keyed into a database and avoiding data entry errors.
We conducted this performance audit from October 2014 to July 2015 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objective.
section_title: Appendix II: Status of Related GAO Recommendations Made Since 2011
paragraphs: Table 10 describes the status of open GAO recommendations associated  with the fiscal year 2005 NDAA’s requirement that we annually assess the  actions taken by the department to comply with its provisions. Since June  2011, we have made 16 recommendations to DOD regarding defense  business systems. As of April 2015, the department had implemented 5,  and partially implemented 11 recommendations. The table also identifies  the category that we assigned to the recommendation to demonstrate its  relationship to the requirements outlined in the act.
section_title: Appendix III: Comments from the Department of Defense
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact above, individuals making contributions to this  report include Michael Holland (assistant director), Camille Chaires, Carl  Barden, Susan Baker, Nabajyoti Barkakati, Wayne Emilien, Nancy  Glover, James Houtz, Monica Perez-Nelson, Stuart Kaufman, Adam  Vodraska, and Shawn Ward.