DOD Business Systems Modernization: Additional Action Needed to Achieve Intended Outcomes
Released: 2015-07-16T00:00:00

Letter

Background

DOD is one of the largest and most complex organizations in the world. In support of its military operations, the department performs an assortment of interrelated and interdependent business functions, including logistics management, procurement, health care management, and financial management. As we have previously reported, the DOD systems environment that supports these business functions is complex and error prone, and is characterized by (1) little standardization across the department, (2) multiple systems performing the same tasks, (3) the same data stored in multiple systems, and (4) the need for data to be manually entered.

For fiscal year 2015, the department requested about $10 billion for its business system investments. According to the department, as of April 2015, its environment includes approximately 2,179 business systems. Of these systems, DOD reports that, for fiscal year 2015, the department approved certification requests for 1,182 business systems covered by the fiscal year 2005 NDAA’s certification and approval requirements. Figure 1 shows how many of these 1,182 covered systems are associated with each functional area.

DOD currently bears responsibility, in whole or in part, for about half (17 of 32) of the programs across the federal government that we have designated as high risk. Seven of these areas are specific to the department, and 10 other high-risk areas are shared with other federal agencies.

Collectively, these high-risk areas in major business operations are linked to the department’s ability to perform its overall mission and affect the readiness and capabilities of U.S. military forces. As such, DOD’s business systems modernization is one of the department’s specific high-risk areas and is essential for addressing many of the department’s other high-risk areas. For example, modernized business systems are integral to the department’s efforts to address its financial, supply chain, and information security management high-risk areas.

The NDAA for Fiscal Year 2005 Requires DOD to Improve Its Management of Defense Business Systems

Congress included provisions in the fiscal year 2005 NDAA, as amended, that are aimed at ensuring DOD’s development of a well-defined business enterprise architecture and associated enterprise transition plan, as well as the establishment and implementation of effective investment management structures and processes. The act requires DOD to, among other things, establish an investment approval and accountability structure along with an investment review process; not obligate funds for a defense business system program with a total cost in excess of $1 million over the period of the current future-years defense program unless the approval authority certifies that the business system program meets specified conditions, including complying with the business enterprise architecture and having appropriate business process reengineering conducted; develop a business enterprise architecture that covers all defense develop an enterprise transition plan for implementing the architecture, and identify systems information in DOD’s annual budget submissions.

The fiscal year 2005 NDAA also requires that the Secretary of Defense submit an annual report to the congressional defense committees on the department’s compliance with these provisions. DOD submitted its most recent annual report to Congress on April 6, 2015, describing steps taken, under way, and planned to address the act’s requirements.

DOD’s Approach to Business Systems Modernization

DOD’s approach to business systems modernization includes reviewing systems annually to ensure that they comply with the fiscal year 2005 NDAA’s business enterprise architecture and business process reengineering requirements. This effort includes both a certification of compliance by lower-level department authorities and an approval of this certification by higher-level department authorities. According to the act, this certification and approval is to occur before systems are granted permission to obligate funds for a given fiscal year. These efforts are to be guided by DOD’s Chief Management Officer (CMO) and Deputy Chief Management Officer (DCMO). Specifically, the CMO’s responsibilities include developing and maintaining a departmentwide strategic plan for business reform; establishing performance goals and measures for improving and evaluating overall economy, efficiency, and effectiveness; and monitoring and measuring the progress of the department.

The DCMO’s responsibilities include recommending to the CMO methodologies and measurement criteria to better synchronize, integrate, and coordinate the business operations to ensure alignment in support of their warfighting mission and developing and maintaining the department’s enterprise architecture for its business mission area. Table 1 describes selected roles and responsibilities and the composition of key governance entities and positions related to business systems modernization as they were documented for the fiscal year 2015 business system certification and approval cycle.

Within the military departments, the entities described in table 1 are supported by portfolio managers who oversee groups of business system investments within specific functional areas. For example, the Department of the Navy’s financial management portfolio manager is responsible for overseeing the Navy’s portfolio of financial management systems.

Overview of DOD’s Integrated Business Framework

In order to manage and oversee the department’s business operations and approximately 1,180 covered defense business systems, the Office of the DCMO developed the Integrated Business Framework. According to officials from the office, this framework is used to align the department’s strategic objectives—laid out in the National Security Strategy, Quadrennial Defense Review, and Strategic Management Plan—with its defense business system investments.

Using the overarching goals of the Strategic Management Plan, principal staff assistants developed six functional strategies that cover nine functional areas. These functional strategies are to define business outcomes, priorities, measures, and standards for a given functional area within DOD. The business objectives and compliance requirements laid out in each functional strategy are to be integrated into the business enterprise architecture.

The precertification authorities in the Air Force, Navy, Army, and other departmental organizations use the functional strategies to guide the development of organizational execution plans, which are to summarize each component’s business strategy for each functional area. Each plan includes a description of how the component’s goals and objectives align with those in the functional strategies and the Strategic Management Plan. In addition, each organizational execution plan includes a portfolio of defense business system investments organized by functional area. The components submit each of these portfolios to the Defense Business Council for certification on an annual basis.

According to the department’s 2015 Congressional Report on Defense Business Operations, for the fiscal year 2015 certification and review cycle, the department empowered the military department chief management officers to manage their business systems portfolios and conduct portfolio reviews. Results were presented to the Defense Business Council and were to address topics such as major improvements and cost reductions, return on investment, risks and challenges, deviations from prior plans, and future goals.

According to DOD’s investment management guidance, for the fiscal year 2015 certification and approval cycle, the Defense Business Council was to review the organizational execution plans and associated portfolios based on four investment criteria—compliance, strategic alignment, utility, and cost—to determine whether or not to recommend the portfolio for certification of funding. The Vice Chairman of the Deputy’s Management Action Group/Defense Business Systems Management Committee was to approve certification decisions and then document the decision in an investment decision memorandum. These memoranda were to indicate whether an individual organizational execution plan has been certified; conditionally certified (i.e., obligation of funds has been certified and approved but may be subject to conditions that restrict the use of funds, a time line for obligation of funds, or mandatory changes to the portfolio of business systems); or not certified (i.e., certification is not approved due to misalignment with strategic direction, mission needs, or other deficiencies).

Overview of DOD’s Business Enterprise Architecture

DOD’s business enterprise architecture is intended to serve as a blueprint for the department’s business transformation efforts. In particular, the architecture is to guide and constrain implementation of interoperable defense business systems by, among other things, documenting the department’s business functions and activities and the business rules, laws, regulations, and policies associated with them. According to DOD, its architecture is being developed using an incremental approach, where each new version of the architecture addresses business mission area gaps or weaknesses based on priorities identified by the department. The department’s business enterprise architecture focuses on documenting information associated with its end-to-end business process areas (e.g., hire-to-retire and procure-to-pay). These end-to-end business process areas may occur across the department’s nine functional areas. For example, hire-to-retire occurs within the human resources management functional area, while the cost management business process area occurs across the acquisition, financial management, human resources management, installations and environment, and logistics and materiel readiness functional areas.

According to DOD officials, the current approach to developing the business enterprise architecture is both a “top down” and “bottom-up” approach. Specifically, the architecture focuses on developing content to support investment management and strategic decision making and oversight (top down) while also responding to department needs associated with supporting system implementation, system integration, and software development (bottom up). Consistent with DOD’s tiered approach to business systems management, the department’s approach to developing its business enterprise architecture involves the development of a federated enterprise architecture, where member architectures (e.g., Air Force, Army, and Navy) conform to an overarching corporate or parent architecture and use a common vocabulary. This approach is to provide governance across all business systems, functions, and activities within the department and improve visibility across the respective efforts.

Overview of DOD’s Business Process Reengineering Efforts

DOD defines business process reengineering as a logical methodology for assessing process weaknesses, identifying gaps, and implementing opportunities to streamline and improve the processes to create a solid foundation for success in changes to the full spectrum of operations. DOD’s reengineering efforts are intended to help the department rationalize its covered business system portfolio, improve its use of performance management, control scope changes, and reduce the cost of fielding business capability. According to DOD officials, the department has taken a holistic approach to business process reengineering, which includes a portfolio and end-to-end perspective. It has also issued business process reengineering guidance that calls for alignment of defense business systems within the Organizational Execution Plan to its functional strategy’s strategic goals.

An important component of the department’s business process reengineering efforts is the problem statement development and review process. A problem statement is developed when a defense business system is seeking certification for a development or modernization effort. The statement is to include, among other things, a description of the problem that the system intends to address and a discussion of the costs, benefits, and risks of various alternatives that were considered. As part of the annual certification and approval process, problem statements are to be reviewed to support that appropriate business process reengineering has been conducted on investments seeking certification.

DOD Has Made Progress Strengthening Its Management of Business Systems, but Needs to Continue Implementing Recommended Improvements

The department has implemented 5 of the 16 recommendations that GAO has made since June 2011 to address each of the overarching provisions for improving business systems management in the fiscal year 2005 NDAA. The fiscal year 2005 NDAA, as amended, includes provisions associated with developing a business enterprise architecture and enterprise transition plan, improving the department’s investment management structures and processes, improving its efforts to certify defense business systems, and mandated budgetary reporting. Since 2011, we have issued four reports in response to the act’s requirement that we assess the actions taken by the department to comply with the In those reports, we have made recommendations to act’s provisions.address each of the act’s overarching provisions for improving business systems management. Table 2 identifies the recommendations we have made since 2011 associated with the fiscal year 2005 NDAA.

Table 3 presents a summary of the current status of these recommendations. Appendix II provides additional information about the status of each recommendation.

As of April 2015, the department had implemented 5 of the 16 recommendations that we have made since June 2011. For example, the department has implemented the recommendation to improve its reporting of business system data in its annual budget request. In particular, the department has established common elements in its three primary repositories used for tracking information about business systems, which allows information about individual business systems to be matched across systems. In addition, the Office of the CIO demonstrated that it conducts periodic data quality assessments. As a result, the department is better positioned to report more reliable information in its annual budget request and to maintain more accurate information about business systems to support its efforts to manage them.

In addition, the department has improved the alignment of its Planning, Programming, Budgeting, and Execution process with its business systems certification and approval process. For example, according to the department’s February 2015 certification and approval guidance, Organization Execution Plans are to include information about certification requests for the upcoming fiscal year as well as over the course of the Future Years Defense Program. As a result, the department’s business system certification and approval process can support better informed decisions about system certifications and inform recommendations on the resources provided to defense business systems as part of the Planning, Programming, Budgeting, and Execution process.

The department has partially implemented the remaining 11 recommendations. For example, the department’s February 2015 investment management guidance, which describes DOD’s business system certification and approval process, identifies four criteria and specifies the associated assessments that are to be conducted when reviewing and evaluating component-level organizational execution plans in order to make a portfolio-based investment decision. The guidance also states that return on investment should be considered when evaluating program cost. However, it does not call for the use of actual- versus-expected performance data and predetermined thresholds.

Further, the Office of the DCMO has developed a draft resource allocation plan for each of its directorates and their respective divisions. This draft plan includes staffing profiles that describe each division’s needed staff competencies and qualifications. However, the Office of the DCMO did not demonstrate that it has addressed other important aspects of strategic human capital planning. For example, the office did not demonstrate that it has developed a skills inventory, needs assessment, gap analysis, and plan to address identified gaps, as called for by our recommendation. Appendix II provides additional information about the recommendations that DOD has fully and partially implemented.

Implementing the remaining 11 recommendations will improve DOD’s modernization management controls and help fulfill the department’s execution of the requirements of the act.

DOD Needs to Improve the Usefulness and Effectiveness of Its Business Enterprise Architecture and Business Process Reengineering Initiatives

DOD’s business enterprise architecture and process reengineering efforts are not fully achieving the intended outcomes described in statute. More specifically, with respect to the architecture, portfolio managers (managers) we surveyed reported that it was generally not effective in achieving its intended outcomes and that its usefulness in achieving benefits, such as reducing the number of applications, was limited. With respect to process reengineering, managers reported these efforts were moderately effective at streamlining business processes, but less so in limiting the need to tailor commercial off-the-shelf systems.

Portfolio managers cited a number of challenges impeding the usefulness and effectiveness of these two initiatives, such as the availability of training, lack of skilled staff, parochialism, and cultural resistance to change. DOD has various improvement efforts under way to address some of these challenges; however, additional work is needed and the managers provided some suggestions for closing the gap. More fully addressing the cited challenges would help increase the utility and effectiveness of these initiatives in driving greater operational efficiencies and savings. Appendix I provides additional details about our survey methodology.

The NDAA for Fiscal Year 2005 Defines Intended Outcomes for the Business Enterprise Architecture

The fiscal year 2005 NDAA, as amended, requires DOD to develop a business enterprise architecture that covers all defense business systems and will be used as a guide for these systems. According to the act, the architecture is intended to help achieve the following outcomes:

Enable DOD to comply with all applicable laws, including federal accounting, financial management, and reporting requirements.

Guide, permit, and constrain the implementation of interoperable defense business systems.

Enable DOD to routinely produce timely, accurate, and reliable business and financial information for management purposes.

Facilitate the integration of budget, accounting, and program information and systems.

Provide for the systematic measurement of performance, including the ability to produce timely, relevant, and reliable cost information.

The act also specifies that the department is not to obligate funds for defense business system programs that have a total cost in excess of $1 million unless the system’s approval authority certifies that the program complies with the business enterprise architecture and the certification is subsequently approved by the department’s Investment Review Board.

Achieving the act’s intended outcomes would contribute to the department’s ability to use the architecture to realize important benefits that we and others have previously identified, such as cost savings or For example, if the architecture effectively guides, permits, avoidance.and constrains the implementation of interoperable systems, that would contribute to increased information sharing and improved system interoperability. As another example, using the architecture to produce timely and reliable business and financial information would contribute to improving management decisions associated with enhanced productivity and improved business and IT alignment, among other things.

The Business Enterprise Architecture Has Produced Limited Value, According to Portfolio Managers

The majority of DOD portfolio managers we surveyed reported that the business enterprise architecture has not been effective in meeting its intended outcomes. More specifically, half of the managers surveyed reported that the business enterprise architecture was effective in enabling compliance with all applicable laws. However, fewer than 40 percent reported that the architecture was effective in helping to achieve the other outcomes called for by the fiscal year 2005 NDAA. Table 4 provides additional information on survey responses regarding the act’s specific requirements.

Portfolio managers provided additional details to further explain their survey responses. Their comments included the following:

The architecture is a standalone effort that does not drive comprehensive portfolio and business management through the various DOD components.

The architecture is overwhelming to review and is not integrated with other activities that occur throughout the remainder of the year.

The compliance requirements are not sufficiently defined to enable system interoperability.

Portfolio managers also reported that the usefulness of DOD’s business enterprise architecture in achieving various potential benefits is limited. For example, 75 percent reported limited achievement of improved change management and 74 percent reported limited achievement of streamlined end-to-end business processes. In addition, 71 percent reported limited achievement of benefits such as a reduced number of applications, improved business and IT alignment, enhanced productivity, and achieving financial benefits such as cost savings or cost avoidance. Table 5 summarizes the portfolio managers’ survey responses.

Although managers reported limited achievement of benefits, two provided specific examples of individual benefits associated with the business enterprise architecture. More specifically, one cited saving $10 million annually due to the establishment of a DOD-wide military housing system that has replaced a number of individual systems. A second reported $11.5 million in architecture-related savings through the retirement of 48 real property and financial management systems.

In addition, officials from the Office of the DCMO provided specific examples of benefits that they stated can be attributed, at least in part, to the department’s business architecture. For example, according to these officials, two proposed new defense business system investments were not approved by DOD due, in part, to architecture reviews that revealed the requested capabilities were already available in existing systems.

Numerous Challenges in Achieving Intended Business Enterprise Architecture Outcomes Identified by Portfolio Managers

The surveyed DOD portfolio managers reported that their functional areas face many challenges in achieving the outcomes described in the NDAA for fiscal year 2005. The most frequently cited challenges reported were the usability of the compliance tool (79 percent), frequent changes to the architecture (75 percent), the availability of training (71 percent), the availability of skilled staff (71 percent), parochialism (67 percent), and cultural resistance to change (63 percent). Table 6 identifies the survey responses to achieving the architecture’s intended outcomes.

Officials from the Office of the DCMO, including the Lead Architect for the business enterprise architecture and the Chief of Portfolio Management, described various efforts under way to address selected challenges identified in our survey results. With regard to the top ranked challenge (usability of DOD’s architecture compliance tool), the office has been working on a more robust replacement tool. As of April 2015, the office had moved architecture content and associated compliance information from its previous tool into its Integrated Business Framework-Data Alignment Portal. Further, the department plans to require all fiscal year 2016 compliance assessments to be completed in this portal environment.

According to officials from the Office of the DCMO, this change will help ensure that architecture-related information is available in the same place, which will help support more sophisticated analysis of information about business systems. For example, by combining information about the architecture, compliance information, functional strategies, and organizational execution plans, the department could more easily conduct analyses that will help support portfolio management. According to these officials, examples of such analyses include the ability to identify the funds certified and approved for various business activities and the ability to identify systems that conduct similar system functions.

With regard to the challenge associated with limited alignment between corporate and component architectures, the officials from the Office of the DCMO stated that they intend to develop an overarching (or federated) architecture that will capture content from, and allow governance across, the department (e.g., Army, Navy, and Air Force). We previously recommended that DOD establish a plan for how it will address business enterprise architecture federation in 2013.

The department’s improvement efforts only address selected reported challenges. However, portfolio managers offered a number of suggestions that relate to other identified challenges that may help close gaps in these efforts. Key suggestions included: Improve tools: Four of 24 managers offered suggestions that relate to compliance tool usability. For example, one portfolio manager stated that functionality should be added to the architecture compliance tool to automatically create and build the architecture artifacts mentioned in compliance guidance using the information already included in the tool for each system. Another portfolio manager stated that there are no tools available that portfolio managers can use to analyze their portfolios relative to the architecture.

Provide additional training: Two managers offered suggestions associated with additional training. For example, one manager reported that the compliance tool is not user friendly and little to no training was offered when programs were required to use it to assert compliance. As a result, this manager added that more training should be made available for using the compliance tool.

Start the process earlier in a system’s life cycle: One manager suggested the architecture be addressed earlier in the acquisition life cycle, such as in the analysis of alternatives phase, in order to help assess whether existing solutions are already employed in other areas of the enterprise. If the architecture compliance process uncovers potential duplication or overlap, it might be easier to stop development of a duplicative system earlier in its life cycle rather than waiting until a business process is more reliant on a planned system that is closer to becoming operational.

Establish priorities: One portfolio manager suggested that the department develop departmental business improvement and integration priorities and develop clearly understandable and verifiable compliance standards that will guide and constrain systems development to help achieve those priorities.

Improve guidance: Two managers suggested that the department improve its guidance to clarify the documentation that systems developed prior to the existence of the business enterprise architecture are required to prepare to address the business enterprise architecture compliance requirement.

Improve content: Seven managers offered suggestions associated with improving content. For example, one manager stated that the business enterprise architecture is large and cumbersome and incomplete in many areas.

Addressing the challenges cited by the portfolio managers could help increase the utility and effectiveness of the department’s business enterprise architecture in driving greater operational efficiencies and cost savings.

The NDAA for Fiscal Year 2005 Defines Intended Outcomes for Business Process Reengineering

The fiscal year 2005 NDAA, as amended, establishes expected outcomes for the department’s business process reengineering efforts. The act states that funds for covered business system programs cannot be certified and approved unless each program’s pre-certification authority has determined that, among other things, appropriate business process reengineering efforts have been undertaken to ensure that the business process supported by the program is, or will be, as streamlined and efficient as practicable and the need to tailor commercial off-the-shelf systems to (a) meet unique requirements, (b) incorporate unique requirements, or (c) incorporate unique interfaces has been eliminated or reduced to the maximum extent practicable. As we have previously reported, modifications to commercial off-the-shelf systems should be avoided to the extent practicable as they can be costly to implement.

Achieving the intended outcomes of the fiscal year 2005 NDAA would increase the department’s ability to realize key benefits to business systems modernization. For example, reengineering business processes to be as streamlined as possible can result in increased efficiencies, a reduced number of interfaces, and decreased program costs.

DOD’s Reengineering Efforts Have Moderately Helped to Streamline Business Processes and Realize Some Efficiencies, but More Work Is Needed to Fully Achieve Intended Outcomes, According to Portfolio Managers

The department’s business process reengineering efforts have had mixed success in achieving their intended outcomes. Specifically, 63 percent of the portfolio managers we surveyed reported that the efforts were effective in helping to ensure that the business processes supported by the defense business systems they manage are (or will be) streamlined and efficient as practicable. As an example, one manager reported this effort highlighted the strengths and weaknesses of the systems within their specific portfolio. Another reported that their portfolio has been reduced from 147 systems to 13 due, in part, to the business process reengineering efforts.

However, the general consensus among surveyed portfolio managers was that the department’s efforts were less effective in helping to limit tailoring of commercial off-the-shelf systems. Only 29 percent reported that DOD’s business process reengineering efforts were effective in eliminating or reducing the need to tailor commercial off-the-shelf systems. Tailoring might be required, for example, because existing policy and guidance might limit a system’s ability to conform to a specific approach for executing a business process that is already built into an individual commercial off-the-shelf system. Another reason given was that managers have limited knowledge about the commercial off-the-shelf products that are available via established enterprise licenses and this limited knowledge makes it difficult to conduct effective business process reengineering.

Table 7 provides additional information on portfolio managers’ responses regarding the effectiveness of DOD’s business process reengineering efforts.

Portfolio managers reported that business process reengineering has been useful in helping to achieve selected benefits. In particular, 70 percent reported that efforts have resulted in streamlined business processes. Sixty-seven percent reported that efforts have resulted in improved documentation of business needs, which is consistent with DOD’s focus on developing problem statements for new capabilities. Such problem statements reflect analysis of a perceived business problem, capability gap, or opportunity. According to officials from the Office of the DCMO, they help ensure that programs are aligned with DOD’s strategic needs, and also assist the department’s efforts in identifying redundancies and duplication. However, only 29 percent of the portfolio managers surveyed reported that efforts to reduce program costs have been effective. Table 8 summarizes the portfolio managers’ survey responses.

The surveyed DOD portfolio managers identified a range of challenges to fully achieving the business process reengineering outcomes described in the fiscal year 2005 NDAA. In particular, cultural resistance to change was the most frequently cited challenge (71 percent), followed by parochialism (i.e., focusing on one’s own sub-organization rather than having an enterprise-wide view.), availability of skilled staff, and availability of training (all at 67 percent). The quality of business process reengineering compliance guidance, the compliance review process, and the timing of the reengineering relative to system development work were also reported as important challenges (all at 63 percent).

Table 9 summarizes survey responses to questions about the challenges to business process reengineering.

DOD has taken steps to improve its reengineering efforts that may, in part, address some of the challenges identified in our survey results. With regard to parochialism (i.e., focusing on one’s own sub-organization rather than having an enterprisewide view), the department is developing online tools that provide additional information to program managers, portfolio managers, pre-certification authorities, and the Defense Business Council. For example, the department’s problem statement portal is to be a repository for problem statement submissions and is to be available departmentwide. In addition, the department has developed its Integrated Business Framework-Data Alignment Portal, which is to provide, among other things, additional information about individual business systems, such as information about which systems execute specific business activities and system functions.

Further, with respect to addressing the challenge associated with the business process reengineering compliance review process, the department has taken steps to help ensure improved accountability for a portion of certification and approval requests. In particular, according to officials from the Office of the DCMO, the DCMO allowed the military departments more autonomy and responsibility for reviewing their system portfolios during fiscal year 2015 certification and approval reviews. Nevertheless, as we have previously reported, and as discussed in appendix II, this process is not guided by specific criteria for elevating certain systems to the Defense Business Council that might require additional oversight.

Notwithstanding these improvement efforts, as reported in feedback by the military department portfolio managers, additional work is needed. These managers provided a number of suggestions to help address the identified challenges. Suggestions included: Improve business process reengineering training: Two portfolio managers offered suggestions that relate to improved training. For example, one manager stated that the department should establish minimum training standards.

Improve business process reengineering guidance: Two managers offered suggestions associated with improved guidance. For example, one portfolio manager stated that sufficient guidance does not exist to describe meaningful business process models or how such models should be analyzed.

Align business process reengineering with system development activities: One portfolio manager stated that the reengineering process should be more closely tied to acquisition milestones instead of being assessed on an annual basis.

According to GAO’s standards for internal controls,ensure that there are adequate means of obtaining information from stakeholders that may have a significant impact on the agency achieving its goals. While we did not evaluate the effectiveness of these suggestions, they may be valuable for the Office of the DCMO to consider in its ongoing and future business process reengineering improvement efforts. More fully addressing the challenges cited by the portfolio managers would help the department achieve better outcomes, including limiting the tailoring of commercial off-the-shelf systems.

Conclusions

DOD has made progress in improving its compliance with section 332 of the NDAA for fiscal year 2005, as amended. Specifically, the department has implemented 5 of the 16 recommendations that we have made since 2011 that are consistent with the requirements of the act and has partially implemented the remaining 11 recommendations. The recommendations not fully implemented relate to improving the department’s investment management processes and efforts to certify defense business systems, among other things. Fully implementing them will help improve DOD’s modernization management controls and fulfill the department’s execution of the act’s requirements.

Collectively, DOD’s business enterprise architecture and business process reengineering efforts show mixed results in their effectiveness and usefulness in achieving the intended outcomes and benefits. Among other things, portfolio managers reported that the architecture does not enable DOD to produce reliable and timely information for decision- making purposes. Additionally, DOD’s reengineering efforts are effective in streamlining business processes, but not in reducing the tailoring of commercial software products. Portfolio managers reported that various challenges exist in achieving intended outcomes and benefits, including cultural resistance, parochialism, and a lack of skilled staff. DOD has various improvement efforts under way; however, gaps exist and portfolio managers provided suggestions on how to close some of them. Until these gaps are addressed, the department’s ability to achieve important outcomes and benefits will continue to be limited.

Recommendation for Executive Action

To help ensure that the department can better achieve business process reengineering and enterprise architecture outcomes and benefits, we recommend that the Secretary of Defense utilize the results of our portfolio manager survey to determine additional actions that can improve the department’s management of its business process reengineering and enterprise architecture activities.

Agency Comments and Our Evaluation

We received written comments on a draft of this report from DOD’s Deputy Chief Management Officer (DCMO). The comments are reprinted in appendix III.

In the comments, the DCMO concurred with our recommendation and stated that the department will use the results of our portfolio manager survey to help make improvements. The DCMO also described associated improvement efforts. For example, the DCMO stated that the department plans to restructure the Business Enterprise Architecture to focus more explicitly on the business processes being executed within the functional domains, which span all levels of the department. DOD officials also provided technical comments, which we have incorporated as appropriate.

We are sending copies of this report to the appropriate congressional committees; the Director, Office of Management and Budget; the Secretary of Defense; and other interested parties. This report also is available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions on matters discussed in this report, please contact me at (202) 512-4456 or chac@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IV.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to (1) assess the actions by the Department of Defense to comply with section 332 of the National Defense Authorization Act (NDAA) for Fiscal Year 2005, as amended and (2) determine the usefulness and effectiveness of DOD’s business enterprise architecture and business process reengineering processes.

To address the first objective, we identified recommendations related to DOD’s business systems modernization efforts that we made in our annual reports from 2011 to 2014 (16 recommendations total) in response to the fiscal year 2005 NDAA’s requirements. Though we have made recommendations in this area prior to 2011, those recommendations have since been closed. We evaluated the department’s written responses and related documentation on steps completed to implement or partially implement the recommendations. Documentation we analyzed included guidance on business enterprise architecture and business process reengineering compliance, guidance on certifying and approving defense business systems, and documentation about the department’s problem statement development and review process. In addition, we interviewed officials from the Office of the Deputy Chief Management Officer and the Office of the Chief Information Officer, and observed a demonstration of the Office of the Deputy Chief Management Officer’s Integrated Business Framework-Data Alignment Portal tool to better understand the actions taken to address our recommendations. We also reviewed the department’s annual report to Congress, which was submitted on April 6, 2015, to identify gaps or inconsistencies with the implementation of the 16 recommendations.

To address our second objective, we determined the intended outcomes of the business enterprise architecture and business process reengineering processes by analyzing the fiscal year 2005 NDAA. We also determined potential benefits associated with the processes by reviewing department guidance on the processes and related documentation. This includes DOD’s business enterprise architecture and business process reengineering guidance, Defense Business System Investment Management Process Guidance, the Business Case Analysis Template, DOD’s Business Enterprise Architecture 10.0 AV-1 Overview and Summary Information, the department’s Strategic Management Plan, and the Information Resource Management Strategic Plan for fiscal years 2014 and 2015. We also reviewed relevant GAO reports on business enterprise architecture and business process reengineering.

We then developed a structured data collection instrument (survey) to gather information on the usefulness of the two specified IT modernization management controls at DOD in achieving their intended outcomes and their effectiveness in achieving associated benefits. As part of this survey, we also developed questions to help us determine (1) challenges related to complying with the processes and (2) suggestions for achieving business enterprise architecture and business process reengineering outcomes, including suggestions for achieving these outcomes in a more cost-effective manner.

Selected questions contained a ratings scale for managers to choose a response that was consistent with the aforementioned topic areas. For example, we asked managers to rate the effectiveness of the business enterprise architecture and business process reengineering efforts using a scale containing the following choices: neither effective nor ineffective, not applicable/no basis to judge.

We also asked managers to identify the extent to which their portfolios had achieved benefits associated with business enterprise architecture and business process reengineering efforts using a scale containing the following choices: little or no extent, or not applicable/no basis to judge.

We pre-tested the questions with various DOD officials including officials from the Office of the Deputy Chief Management Officer, and with portfolio and program-level officials within the military departments. As a result, we determined that the military department portfolio managers were in the best position to answer our questions because they manage and have a perspective across an entire portfolio of defense business systems. Officials from DCMO’s Management, Policy, and Analysis Directorate provided us with a list of portfolio managers for the three military departments. We did not include portfolio managers for DOD entities outside of the military departments.

We obtained responses from all surveyed portfolio managers (24 in total). Accordingly, these results are generalizable. We analyzed and summarized the survey results to help determine the usefulness and effectiveness of DOD’s business process reengineering and enterprise architecture efforts, as well as related challenges and suggestions for improvement. In addition, though we collected examples of cost savings estimates from managers, and cite them in the report, we did not assess the cited cost savings estimates.

We also met with managers of selected DOD business system programs and other knowledgeable DOD officials to discuss their perspectives on DOD’s business enterprise architecture and business process reengineering efforts. This included interviewing officials associated with defense business programs from each of the military departments and from across various business functions, including program managers, enterprise architects, and other technical and program operations officials. Further, when available, we reviewed documentation provided by DOD program managers to substantiate answers provided as part of our interviews. We also discussed the survey results with officials from the Office of the DCMO to obtain their perspectives on the results and discussed with these officials ongoing efforts to improve the department’s business process reengineering and enterprise architecture efforts.

The practical difficulties of conducting any survey may introduce errors, commonly referred to as nonsampling errors. For example, difficulties in how a particular question is interpreted, in the sources of information that are available to respondents, or in how the survey data are analyzed can all introduce unwanted variability into survey results. To minimize such nonsampling errors, a social science survey specialist designed the questionnaire in collaboration with GAO staff with subject matter expertise. As stated earlier, the questionnaire was pre-tested to ensure that the questions were relevant, clearly stated, and easy to comprehend. When data from the survey were analyzed, an independent analyst reviewed the computer program used for the analysis of the survey data. Since this was a web-based survey, respondents entered their answers directly into the electronic questionnaire, thereby eliminating the need to have the data keyed into a database and avoiding data entry errors.

We conducted this performance audit from October 2014 to July 2015 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objective.

Appendix II: Status of Related GAO Recommendations Made Since 2011

Table 10 describes the status of open GAO recommendations associated with the fiscal year 2005 NDAA’s requirement that we annually assess the actions taken by the department to comply with its provisions. Since June 2011, we have made 16 recommendations to DOD regarding defense business systems. As of April 2015, the department had implemented 5, and partially implemented 11 recommendations. The table also identifies the category that we assigned to the recommendation to demonstrate its relationship to the requirements outlined in the act.

Appendix III: Comments from the Department of Defense

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact above, individuals making contributions to this report include Michael Holland (assistant director), Camille Chaires, Carl Barden, Susan Baker, Nabajyoti Barkakati, Wayne Emilien, Nancy Glover, James Houtz, Monica Perez-Nelson, Stuart Kaufman, Adam Vodraska, and Shawn Ward.