Defense Efficiencies: Action Needed to Improve Evaluation of Initiatives
Released: 2014-01-17T00:00:00

Letter

Background

As part of its fiscal year 2012 budget request, DOD outlined estimated savings of about $178 billion to be realized over a 5-year time period beginning in fiscal year 2012. According to DOD, these savings included about $154 billion from the Secretary’s initiative and about $24 billion from other sources. Specifically,

The military departments and SOCOM identified a total of $100 billion in savings as a result of their efforts to support the Secretary’s initiative. A majority of the projected savings identified by the military departments and SOCOM (approximately $70 billion, or 70 percent) was planned to be reinvested in high-priority military needs—such as enhancing weapon systems—while the remainder was planned to be used to address operating costs resulting from areas such as health care and training. In addition to the $100 billion from the military departments and SOCOM, DOD proposed a $78 billion reduction in its overall budget plan over a 5-year time period, covering fiscal years 2012 through 2016, which reflected a 2.6 percent reduction from DOD’s fiscal year 2011 budget submission over the same time period. Of this amount, $54 billion reflected projected savings identified from a health care policy assessment, government-wide civilian pay freeze, and other specific areas identified by the Secretary where immediate action could be taken department-wide. The remaining $24 billion reflected revised economic assumptions, projected savings from restructuring the Joint Strike Fighter weapon program, and projected savings from reducing the size of the Army and Marine Corps.

Information accompanying DOD’s fiscal year 2012 budget request catalogued the $100 billion in savings from the military departments and SOCOM under the following four categories: reorganizations, such as restructuring headquarters management and eliminating unneeded task forces; better business practices, such as reducing energy consumption; program reductions and terminations, such as terminating weapon system programs; and reductions in lower priority programs, such as shifting funding requests from military construction projects to base operations.

Table 1 shows the specific amounts of projected savings reported for each category and military department.

Among other things, the the budget account from which each savings identified will be derived; the number of military personnel and full-time civilian employees of the federal government affected by such savings; the estimated reductions in the number and funding of contractor personnel caused by such savings; a specific description of activities or services that will be affected by such savings, including the locations of such activities or services; and certain information regarding each reinvestment planned to be funded with efficiency initiative savings.

In June 2012, DOD submitted its report to the congressional defense committees and provided some information on the categories above. Among other things, at that time, DOD reported that it was on track to meet estimated savings targets for all of its efficiency initiatives. However, the report did not include a comprehensive analysis of reinvestments because, according to DOD, many areas in which reinvestments had occurred due to the efficiency initiatives included in the fiscal year 2012 President’s Budget request had been offset by major force structure changes and other reductions in its fiscal year 2013 budget request. In briefings to the Comptroller delivered in February 2013 and March 2013, the military departments and SOCOM reported that they remain, with a few exceptions, on track to meet original savings estimates associated with their individual efficiency initiatives. At the time of our review, DOD had not compiled DOD-level summary information on its progress in achieving its original savings estimate of $100 billion. DOD officials cited various reasons why compiling and reporting on this information may not be feasible. For example, they noted that the need to apply spending reductions in response to sequestration affected funding levels for many programs, including areas targeted for efficiency initiatives. As a result, DOD had to adjust plans for executing programs as well as for implementing initiatives, such as adjusting the scope of initiatives or the timing of actual or planned actions for implementation. Because of the variability in its programs and funding amounts, DOD officials stated that, at a certain point, it becomes difficult to isolate whether savings were achieved solely due to implementing initiatives rather than a combination of factors.

We have previously reported on opportunities for DOD to improve tracking and reporting on cost savings and efficiencies. For example, in March 2012, we reported that DOD took steps to examine its headquarters resources for potential efficiencies, but that it faced an underlying challenge of not having complete and reliable headquarters information available to make related assessments and decisions. To improve DOD’s ability to identify how many headquarters personnel it has, we recommended that the Secretary of Defense direct the Director of Administration and Management, in consultation with the Under Secretary of Defense for Personnel and Readiness, to revise its DOD Instruction 5100.73, Major DOD Headquarters Activities, to, among other things, include all major DOD headquarters activity organizations. DOD partially concurred with our recommendation and commented that the shortcomings in the instruction have limited impact on the management of the department. In July 2012, we reported that, as part of one of its efficiency initiatives, the Air Force estimated it could save about $1.7 billion in its training program by reducing live flying hours and taking other steps, such as increasing the use of virtual training, but lacked a methodology for determining the costs of virtual training and therefore, did not consider these costs in its estimate.visibility over the costs related to virtual training, we recommended that the Secretary of Defense direct the Secretary of the Air Force to develop a methodology for collecting and tracking cost data for virtual training and use this cost data to help inform future decisions regarding the mix of live and virtual training. DOD concurred with our recommendation and To improve decision makers’ identified actions being taken to enhance its ability to capture costs related to virtual training.

Additionally, in December 2012, we reported that DOD had developed an approach for the military departments and SOCOM to follow in tracking and reporting on the status of efficiency initiatives; however, DOD’s approach had some limitations that resulted in incomplete reporting which could limit the visibility of senior leaders in monitoring progress toward achieving programmatic and financial goals. Specifically, the offices of the Comptroller and DCMO had provided general direction through emails, briefings, and training, and, according to officials, had given the military departments and SOCOM flexibility to report on the efficiency initiatives that they felt were most important. In practice, the Army, Air Force, and SOCOM had reported on all of their efficiency initiatives, while the Navy reported on a subset of its initiatives based on what it deemed to be at medium or high risk of experiencing implementation issues or adversely affecting the Navy’s ability to carry out its mission. With respect to realized savings, we reported that the military departments and SOCOM told us they were on track to realize estimated savings, but found some instances where certain costs were not considered. For example, for its initiative to reduce fleet shore command personnel from U.S. Pacific Fleet and U.S. Fleet Forces Command, the Navy did not account for potential increases in relocation costs for moving personnel to other areas within the Navy. We found that the military departments and SOCOM were not reporting consistent information or complete cost information because they had not received written guidance with standardized definitions and methodologies. Rather, the direction provided by the offices of the Comptroller and DCMO did not specify whether all of the costs associated with implementing an efficiency initiative, including costs not initially identified, should be included. To ensure more complete and consistent reporting, we recommended that DOD develop guidance with standardized definitions and methodologies for the military departments and SOCOM to use in reporting. Further, we recommended that guidance should define reporting requirements for such things as the specific types of costs associated with implementing the initiatives, including implementation costs that were not initially identified in calculations of net savings. DOD agreed with the spirit and intent of our recommendation and indicated it planned to issue additional formal guidance in the February 2013 timeframe. The status of DOD’s implementation of this recommendation is discussed in more detail later in this report.

Since initiating its initial round of initiatives for fiscal year 2012, DOD has continued to identify and implement efficiency initiatives. Specifically, in information accompanying its fiscal years 2013 and 2014 budget requests, DOD identified additional efficiency initiatives, referred to as More Disciplined Use of Resources (MDUR) initiatives. These initiatives are expected to generate $60 billion in savings for the period of fiscal years 2013 through 2017 and an additional $34 billion for the period of fiscal years 2014 through 2018. While savings generated by the Secretary’s fiscal year 2012 efficiency initiatives were to be reinvested, savings from the MDUR initiatives were intended to help the department meet reductions to its budget, and therefore will not be reinvested. More recently, on July 31, 2013, as part of DOD’s Strategic Choices and Management Review, the Secretary announced his plan to implement an additional round of efficiency initiatives in fiscal year 2015. According to the Secretary, a tenet of the review was the need to maximize savings from reducing DOD’s overhead, administrative costs, and other institutional expenses. These initiatives would include management reforms, coupled with consolidations, personnel cuts, and spending reductions that would reduce DOD’s overhead and operating costs by some $10 billion over the next 5 years and almost $40 billion over the next decade.

DOD Has Taken Steps to Further Refine Its Approach to Track and Report on Its Efficiency Initiatives

DOD has taken steps to further refine its approach to its tracking and reporting on the implementation of its efficiency initiatives. Specifically, DOD issued written guidance that standardizes and expands the type of information on efficiency initiatives that the military departments and SOCOM are expected to report, which may improve visibility on the progress and risks in implementation for DOD decision makers. Moreover, in commenting on a draft of this report, DOD stated that it will cease to track initiatives once they have been implemented and will select for detailed tracking only those initiatives where this information will help it manage more effectively.

Following our December 2012 report, the Comptroller issued written guidance in February 2013 establishing a standardized format for reporting on the fiscal year 2012 efficiency initiatives as well as the fiscal year 2013 MDUR initiatives. According to DOD officials, this guidance is also applicable to initiatives identified in fiscal year 2014 and any future initiatives. In contrast to the way they reported before, the military departments and SOCOM were now expected to report consistently and provide the status of their efficiency initiatives, including summary information related to (1) whether original net savings projections across the Future Years Defense Program are being met, (2) risks to program(s), mission(s), or resources associated with the efficiency initiative (characterized as “low”, “medium”, or “high” risk), and (3) any risks to “milestones” or the implementation status of the efficiency initiative (e.g., characterized as “on track,” “off track but can meet major milestones,” or “off track and cannot meet major milestones”). Only in instances where the military departments and SOCOM identified programs that were not achieving original net savings estimates or where program or milestone risk had been identified, the guidance requires further detail, including how implementation would be achieved. Further, all of the information was to be reported in a manner that mirrored the descriptions contained in DOD’s fiscal year 2012 budget request justification book for the efficiency initiatives, whereby some of the efficiency initiatives were collapsed into broader groups of initiatives referred to by descriptive titles. In February and March 2013, using the new February 2013 guidance, the military departments and SOCOM completed the first round of semi-annual reporting on the fiscal year 2012 efficiency initiatives and fiscal year 2013 MDUR initiatives.

In reviewing the military departments’ and SOCOM’s February 2013 and March 2013 reports, we observed that, consistent with the aforementioned February 2013 guidance, the military departments and SOCOM reported details associated with only those efficiency initiatives that were not achieving original net savings estimates or where program or milestone risk had been identified. As a result, detailed information on the full range of efficiency initiatives and related programs was not included in their reports. For example, the Air Force has as many as 10 individual initiatives that comprise its acquisition management initiative. Absent a requirement in the February 2013 guidance to report on each of those underlying initiatives, DOD decision makers would only receive information on the overall acquisition management initiative. Moreover, as a result of the reporting direction, DOD decision makers would receive detailed information on the overall acquisition management initiative only if the initiative is not meeting original savings estimates, or where program or milestone risk had been identified. Prior to the February 2013 guidance, some departments and SOCOM had previously chosen to report on all their initiatives. In reviewing the reports developed by the military departments and SOCOM in February 2013 and March 2013, we observed that information on all initiatives was now unavailable to DOD decision makers, thus hindering their ability to assess implementation progress across the full range of efficiencies. We discussed with DOD Comptroller officials whether reporting on only those efficiency initiatives not achieving their original estimates or facing risk had provided the Comptroller with sufficient details to oversee all of the initiatives. Comptroller officials agreed that reporting on each of the individual efficiency initiatives would improve DOD decision makers’ visibility and therefore provide information needed for their oversight. They also noted that it would facilitate DOD’s ability to address any future congressional reporting requirements. As a result, the Comptroller’s office subsequently issued guidance in October 2013 that, according to these officials, superseded the February 2013 guidance and expanded the amount of information to be reported. Specifically, this guidance directed the military departments and SOCOM to submit further detail for all efficiency initiatives, rather than merely those not achieving the original estimates or at risk. Beginning in October 2013, the military departments and SOCOM began submitting reports that included this broader set of information.

While obtaining this broader set of information, DOD stated in its written comments on a draft of this report, provided on January 6, 2014, that it will narrow the scope of efficiency initiatives that will be tracked due to the period of constrained resources it is experiencing. DOD stated that it will cease tracking initiatives once they have been implemented, and will select for detailed tracking only those initiatives where this information will help it manage more effectively. In clarifying its written comments, DOD officials stated that all of its efficiency initiatives, except those implemented or which strictly call for terminations of programs, such as weapons systems, will be selected for detailed tracking. While the October 2013 guidance, which requires the military departments and SOCOM to report more detailed information on the full range of ongoing efficiency initiatives, does not specify that initiatives that strictly call for program terminations should no longer be tracked, we believe it is reasonable for DOD to cease the tracking of initiatives that strictly call for program terminations.

We note that, in issuing additional guidance on its tracking and reporting on efficiency initiatives, DOD did not include any direction as to the specific types of costs that the military departments and SOCOM should consider in determining realized savings associated with implementation, such as costs that were not initially identified in calculations of net savings, as we had recommended in our December 2012 report. According to a Comptroller official, DOD has various guidance on developing cost estimates that the military departments and SOCOM can use in determining savings associated with the implementation of their efficiency initiatives. We reviewed the documents and discussed with the efficiency initiative focal points how, or if, this guidance was applied in developing their cost estimates. Some of the program managers with whom we spoke confirmed that while they were aware of existing guidance on developing cost estimates, they had not been instructed to use this guidance to determine specific types of costs that should be considered in calculating net savings. As a result, we continue to believe that our prior recommendation in our December 2012 report has merit and should be implemented.

Military Departments and SOCOM Vary in the Extent and Nature of Their Efforts to Evaluate the Impact of Their Efficiency Initiatives

The military departments and SOCOM have taken steps to evaluate some of their efficiency initiatives, such as establishing performance measures and collecting performance data. However, these efforts have largely occurred on an ad hoc basis and vary by efficiency initiative because DOD has not established a requirement for performing such evaluations. As a result, DOD lacks a systematic basis for evaluating the impact of its efficiency initiatives on improving program efficiency or effectiveness.

In setting forth the initial efficiency initiatives, the Secretary of Defense intended for DOD to improve the effectiveness and efficiency of its programs and activities. The Secretary also directed that any efficiency initiative must be specific, actionable, and measurable. Our prior review of federal agencies’ efficiency efforts concluded that an improvement in efficiency need not only involve a reduction in costs, but also can be achieved by maintaining federal government services or outcomes with fewer resources (such as time or money), or improving or increasing the quality or quantity of services while maintaining (or reducing) resources. In addition, we and other agencies, such as the Office of Management and Budget (OMB), have documented the need to develop performance measures for evaluating progress toward achieving desired outcomes. For example, as we have previously concluded, performance measures should be measurable, outcome-oriented, and actively tracked and reported. As our prior work has shown, leading organizations that employ result-oriented management use performance information as a basis for decision making and have found this approach improves program results.

As previously discussed, the Comptroller’s October 2013 guidance provides direction on how the military departments and SOCOM are to approach reporting on the status of their efficiency initiatives, but does not require them to develop approaches for evaluating the impact of initiatives on achieving desired outcomes. In practice, we found that the military departments and SOCOM varied in the extent to which they evaluated individual efficiency initiatives, including whether they had established measures or indicators to gauge the impact on program efficiency or effectiveness beyond savings. The following paragraphs provide examples of the services’ and SOCOM’s efforts to evaluate certain efficiency initiatives.

Air Force’s Facility Sustainment Initiative: This initiative is intended to reduce infrastructure maintenance costs by a total of $1.4 billion during the period of fiscal years 2012 through 2016 by performing preventative maintenance before critical failures occur. The Air Force uses a model to predict and prioritize infrastructure most at risk for critical failures and then focuses preventive maintenance efforts on such infrastructure. Furthermore, the Air Force has established measures to track the amount of hours spent performing preventive and corrective maintenance over the course of the initiative to determine whether this effort achieved the intended outcome, which was to reduce the amount of more costly corrective maintenance performed. We have previously concluded that deferring facility sustainment can lead to shortened facility service lives and increased future costs for recapitalization.

SOCOM’s Information Technology Services Efficiency Initiative: SOCOM established a new approach for its information technology services that is intended to reduce costs by a total of $394 million during the period of fiscal years 2012 through 2016. According to SOCOM officials, the new approach involved a contract framework for information technology services that reduces costs by awarding funds directly to the organizations that provide the services on a competitive basis, rather than through an intermediary that selects the organizations that provide the information technology services. The approach also adopts other best practices for procurement, such as providing performance-based incentives. As part of this initiative, SOCOM established multiple individual measures to assess contractor performance, such as answering a help desk call within a set amount of time or tracking trends on resolving information technology issues such as user access, but does not have measures in place to evaluate how the overall impact of the initiative affects the delivery of information technology services relative to the previous approach. SOCOM officials explained that because the implementation of its Information Technology Services Efficiency Contract occurred prior to it being identified as an efficiency initiative, methods of evaluating its effectiveness or efficiency, other than cost, were not established.

Navy’s Total Ownership Cost Initiative: This initiative seeks to achieve efficiencies through life cycle management of the Navy’s ships and encompasses multiple underlying initiatives, such as the Navy’s Revised Virginia Class Drawings Efficiency Initiative. This initiative is intended to reduce costs of $30.3 million during the period of fiscal years 2012 through 2016 by moving away from reliance on paper documents toward an electronic system that allows multiple users to make revisions and access up-to-date documents. The Navy has not yet identified measures to evaluate how increasing the use of the electronic system to process technical documents will maintain or improve work processes. Navy officials indicated the same is true for other fiscal year 2012 initiatives that make up the Navy’s Total Ownership Cost initiative and commented that the focus was on tracking savings and not on developing efficiency measures to assess whether its initiatives, once implemented, improve the effectiveness or efficiency of these programs.

Army and Air Force Data Center Consolidation Efficiency Initiatives: These initiatives are part of the larger Federal Data Center Consolidation Initiative, directed by OMB, that seeks to consolidate information technology infrastructure and activities to save energy costs, among other goals, and are expected to reduce the Army and Air Force’s costs by $490 million and $180 million, respectively, during fiscal years 2012 through 2016. Both the Army and Air Force have taken steps to implement these initiatives. According to Air Force officials, they had begun to establish measures that could be used to evaluate the impact of these initiatives, but faced some challenges due to changing guidance. For example, officials discussed that OMB guidance issued after the initiatives were underway expanded the definition of a data center and effectively increased the scope of the military departments’ consolidation effort after these initiatives were submitted in the fiscal year 2012 budget request. This resulted in the reconfiguration of planning and implementation schedules. Air Force officials also stated that they had begun to develop performance measures to assess impact when a new DOD effort to establish a secure, joint information environment was put in place. Therefore, the Air Force had to adjust their implementation plans and postpone the development of their measures to ensure actions taken on this initiative conformed to the new DOD requirement.

While the focus of DOD’s effort was to quickly identify funds that could be reinvested into other higher priority programs, military department and SOCOM officials explained that because the initial effort to identify efficiency initiatives occurred late at the end of the cycle they used to build their fiscal year 2012 budget submission, their effort focused on tracking savings targets and not on developing measures to evaluate impact. In the subsequent budget cycles that included the MDUR initiatives, the focus remained on identifying areas for reductions in spending. DOD officials agreed that additional measures could be useful to evaluate impacts—beyond savings—of their efficiency initiatives. Our prior work concluded that such measures can assist managers in determining whether desired outcomes were being achieved or if adjustments were needed, such as in the scope of the initiative or to the nature or timing of implementation actions. Without a systematic way to evaluate the impact of its efficiency initiatives, DOD is limited in its ability to assess whether the efficiency initiatives have improved the effectiveness or efficiency of its programs and activities.

Conclusions

Over the past few years, in light of mounting fiscal pressures, DOD has continued to identify and implement efficiency initiatives with certain goals in mind, including achieving cost savings and seeking opportunities to enhance the efficiency or effectiveness of its programs and activities. DOD’s recent efforts to refine its approach for tracking and reporting on its current efficiency initiatives has the potential for providing greater oversight to decision makers on progress of the military departments and SOCOM on the status of their implementation efforts. However, its efforts to date do not sufficiently ensure that leaders have the information they need to fully assess the impact these initiatives are having on DOD’s programs and activities. Having a systematic way to evaluate the impact of its efficiency initiatives beyond cost savings could provide DOD the ability to determine whether or not its initiatives are improving the efficiency and effectiveness of its programs and activities while also achieving savings. Such information could also inform DOD as to whether actions are needed to make adjustments to the scope of any given initiative and related programmatic actions necessary for implementation.

Recommendation for Executive Action

To enhance DOD’s ability to determine whether its efficiency initiatives are having the desired effect of improving efficiency and effectiveness, we recommend that the Secretary of Defense require the military departments and SOCOM to develop approaches for evaluating the impact of their efficiency initiatives, such as establishing performance measures or other indicators, collecting related performance information, and using this information to measure progress in achieving intended outcomes associated with their initiatives until implemented.

Agency Comments and Our Evaluation

In written comments on a draft of this report, DOD concurred with our recommendation and provided additional comment. Specifically, DOD concurred with having the military departments and SOCOM develop performance measures and other indicators for evaluating the impact of its efficiency initiatives, and commented that it has decided to cease tracking initiatives once they have been implemented. DOD also provided technical comments, which were incorporated as appropriate. The full text of DOD’s comments is reprinted in appendix II.

In its overall comments, DOD stated it intends to continue to refine its procedures, guidance, and oversight in order to achieve its goal of identifying and implementing efficiencies. DOD also stated that during this period of constrained resources, it must avoid creating a costly and redundant oversight process. To that end, DOD stated that it will cease tracking initiatives once they have been implemented, and will select for detailed tracking only those initiatives where this information will help it manage more effectively. We have modified the report to reflect DOD’s decision.

In clarifying its written comments, DOD officials stated that DOD will select for detailed tracking all of its efficiency initiatives except those implemented or which strictly call for terminations of programs, such as weapons systems. While the October 2013 guidance does not specify that initiatives that strictly call for program terminations should no longer be tracked, in a resource-constrained environment, we believe it is reasonable for DOD to do so. We also expect that DOD will clarify this revised approach in any future guidance to the military departments and SOCOM.

In addition, we have modified the recommendation to clarify its intent that DOD should develop an approach for evaluating the impact of its efficiency initiatives until those initiatives have been implemented.

We are sending copies of this report to the Secretary of Defense and appropriate congressional committees. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-5257 or merrittz@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix III.

Appendix I: Scope and Methodology

To determine the progress DOD has made in adjusting its approach to tracking and reporting on the implementation of its efficiency initiatives since we last reported in December 2012 and to assess the extent to which DOD is evaluating the impact of its efficiency initiatives on DOD programs and activities, we reviewed guidance and documentation issued at the department-wide level as well as within the military departments and SOCOM. We also interviewed officials from the offices of the Comptroller and DCMO, the military departments, and SOCOM who are involved in monitoring the implementation of its efficiency initiatives to discuss their approach to tracking and reporting on the initiatives. Specifically, we obtained available information from each of the military departments and SOCOM, including briefings prepared for senior DOD officials, on the current status of initiatives, how original estimates of savings compared with savings realized to date, and any program or timeline risks associated with implementing the efficiency initiatives. Additionally, we reviewed existing guidance to identify any requirements for evaluation of the efficiency initiatives. We then analyzed information provided by each of the military departments and SOCOM as well as interviewed officials from each of the military services and SOCOM serving as focal points for the efficiency initiatives to determine the expected outcome or impact for individual initiatives, and what steps they have taken to evaluate the impact of efficiency initiatives.

We conducted this performance audit from May 2013 through January 2014 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Department of Defense

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Tina Won Sherman, Assistant Director; Grace Coleman; Susan Langley; Ricardo A. Marquez; Sharon L. Pickup; Mike Silver; Michael Shaughnessy; Susan Tindall; and Sarah Veale made key contributions to this report.