Defense Efficiencies: Action Needed to Improve Evaluation of Initiatives
Published: 2014-01-17T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: In May 2010, the Secretary of Defense announced a department-wide initiative with the goal of achieving efficiencies and reducing excess overhead costs while reinvesting those savings in sustaining DOD's force structure and modernizing its weapons portfolio. The Secretary tasked the military departments and SOCOM to find estimated savings of about $100 billion over the period of fiscal years 2012 to 2016. For fiscal years 2013 and 2014, DOD identified additional efficiency initiatives. The National Defense Authorization Act for Fiscal Year 2012 mandated that GAO assess the extent to which DOD has tracked and realized savings proposed pursuant to the initiative to identify $100 billion in efficiencies. As the second report in response to this mandate, this report addresses 1) DOD's progress in adjusting its approach to tracking and reporting on the implementation of its efficiency initiatives since GAO's December 2012 report, and 2) the extent to which DOD is evaluating the impact of its initiatives. GAO reviewed guidance, and analyzed and discussed information developed after December 2012 with DOD officials.
section_title: What GAO Found
paragraphs: The Department of Defense (DOD) has refined its approach for tracking and reporting on the status of efficiency initiatives by establishing specific requirements to standardize and expand the type of information that the military departments (Army, Navy, and Air Force) and U.S. Special Operations Command (SOCOM) report to senior decision makers. Initially, DOD provided general direction through emails, briefings, and training, which gave the military departments and SOCOM flexibility to selectively report on the initiatives that they believed were important, resulting in inconsistencies. For example, prior to February 2013, all but the Navy had chosen to report on all their initiatives. In February 2013, the DOD Comptroller issued written guidance that specified the type of information to be reported, including 1) whether original net savings projections are being met, and 2) any associated program or milestone risks. In instances where original net savings projects were not met or risks were identified, the guidance required further detail such as how implementation would be achieved. As a result, in their March 2013 reports, the military departments and SOCOM only reported details on those initiatives that were not achieving original net savings estimates or where risk had been identified. GAO observed, during this review, that information on all initiatives was now unavailable to DOD decision makers, thus hindering their ability to assess implementation progress across the full range of initiatives. Comptroller officials agreed that such information would enhance DOD's oversight, and in October 2013, the DOD Comptroller issued updated guidance, directing that this information also be reported on initiatives on track to achieve savings or not experiencing risk. The military departments and SOCOM subsequently began submitting reports with this broader set of information.
The military departments and SOCOM have taken steps to evaluate the impact of some of their efficiency initiatives, such as establishing performance measures to assess their impact on achieving desired outcomes. However, this has largely occurred on an ad hoc basis and varies by initiative because DOD has not required such evaluations. As a result, DOD lacks a systematic basis for evaluating whether its various initiatives have improved the efficiency or effectiveness of its programs or activities. In setting forth initiatives, the Secretary of Defense intended for DOD to improve the effectiveness and efficiency of its programs and activities, and that related initiatives should be specific, actionable, and measurable. While DOD has provided direction on how the military departments and SOCOM are to report on implementation status, this direction does not require them to develop approaches for evaluating the impact of their initiatives. In practice, the military departments and SOCOM varied in the extent to which they evaluated initiatives, including whether they had established measures or other indicators to assess outcomes. For example, GAO found instances where the military departments and SOCOM had established measures and assessed progress for some but not all initiatives. Developing a more systematic approach to evaluating the impact of its initiatives could provide DOD with more complete information to assess whether the initiatives are accomplishing desired outcomes, beyond achieving savings, and whether adjustments are needed in the scope of implementing the initiatives.
section_title: What GAO Recommends
paragraphs: GAO recommends that DOD establish a requirement for the military departments and SOCOM to develop approaches for evaluating the impact of their efficiency initiatives, such as developing performance measures or other indicators. DOD concurred with GAO's recommendation, and provided additional comments that it will cease tracking initiatives that strictly call for program terminations. GAO believes this to be a reasonable approach.

section_title: Letter
paragraphs: In May 2010, the Secretary of Defense announced that the Department  of Defense (DOD) was to undertake a department-wide initiative to  assess how the department is staffed, organized, and operated with the  goal of reducing excess overhead costs, and reinvesting those savings in  sustaining DOD’s current force structure and modernizing its weapons  portfolio. The Secretary’s initiative targeted both short- and long-term  improvements and set specific goals and targets for achieving savings  and efficiencies, which are expected to be achieved from fiscal years  2012 through 2016. As part of this effort, the Secretary of Defense tasked  the military departments (Army, Navy, and Air Force) and other  components including U.S. Special Operations Command (SOCOM) to  find savings of about $100 billion in overhead costs over the specified  time period. About one-third of these savings would be used to fund  higher-than-expected operating costs and the remaining two-thirds—over  $70 billion—would be reinvested in high-priority military capabilities in  fiscal years 2012 through 2016. Of this amount, about $11 billion was  projected to be achieved in fiscal year 2012 and available for  reinvestment, and about $14 billion was projected to be achieved and  available for reinvestment in fiscal year 2013.
Information accompanying DOD’s fiscal year 2012 budget request  outlined specific efficiency initiatives identified by the military departments  and SOCOM. Prior to the beginning of fiscal year 2012, DOD’s  Comptroller and Deputy Chief Management Officer (DCMO) began  developing an approach for the military departments and SOCOM to track  and report on their efforts to implement efficiency initiatives and realize  savings.
The National Defense Authorization Act for Fiscal Year 2012 mandated  GAO to assess the extent to which DOD has tracked and realized the  savings proposed pursuant to the initiative to identify at least $100 billion  in efficiencies during fiscal years 2012 through 2016, and to submit a  report annually for each of those fiscal years. In December 2012, we  issued our first report in response to this requirement. We reported that  DOD had developed an approach to track and report on the  implementation of its efficiency initiatives, but the approach had some  limitations affecting its ability to monitor progress toward achieving  programmatic and financial goals. To ensure more complete and  consistent reporting, we recommended that DOD develop guidance that  would standardize definitions and methodologies for DOD components to  use in reporting their efficiency initiatives and savings, including defining  specific types of costs associated with implementing its initiatives. Since  then, DOD has continued to refine its approach to monitoring  implementation of its efficiency initiatives, including issuing additional  guidance clarifying requirements for tracking and reporting. In addition,  DOD included additional efficiency initiatives in information accompanying  its fiscal years 2013 and 2014 budget requests, and has announced its  intent to implement an additional round of efficiency initiatives in fiscal  year 2015.
This report addresses 1) DOD’s progress in adjusting its approach to  tracking and reporting on the implementation of its efficiency initiatives  since we issued our December 2012 report; and 2) the extent to which  DOD is evaluating the impact of its initiatives on DOD’s programs and  activities. To perform our work, we reviewed guidance and documentation  issued at the department-wide level as well as within the military  departments and SOCOM to identify adjustments to tracking and  reporting requirements since our December 2012 report. We also  interviewed officials from the offices of the Comptroller and DCMO, the  military departments, and SOCOM who are involved in monitoring the  implementation of its efficiency initiatives to discuss their approach to  tracking and reporting on the initiatives. Specifically, we obtained  available information from each of the military departments and SOCOM,  including briefings prepared for DOD officials, on the current status of  efficiency initiatives, how original estimates of savings compared with  savings realized to date, and any program or timeline risks associated  with implementing the initiatives. Additionally, we reviewed existing  guidance to identify any requirements for the evaluation of efficiency  initiatives. We then analyzed information provided by each of the military  departments and SOCOM as well as interviewed officials serving as focal  points to determine the expected outcome or impact for individual  initiatives, and what steps they have taken to evaluate the impact of those  efficiency initiatives. For example, based on discussions with officials, we  identified examples of initiatives where performance measures or other  indicators had or had not been established, but did not evaluate the  application or appropriateness of the measures or other indicators.
We conducted this performance audit from May 2013 through January  2014 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: As part of its fiscal year 2012 budget request, DOD outlined estimated  savings of about $178 billion to be realized over a 5-year time period  beginning in fiscal year 2012. According to DOD, these savings included  about $154 billion from the Secretary’s initiative and about $24 billion  from other sources. Specifically,
The military departments and SOCOM identified a total of $100 billion  in savings as a result of their efforts to support the Secretary’s  initiative. A majority of the projected savings identified by the military  departments and SOCOM (approximately $70 billion, or 70 percent)  was planned to be reinvested in high-priority military needs—such as  enhancing weapon systems—while the remainder was planned to be  used to address operating costs resulting from areas such as health  care and training.  In addition to the $100 billion from the military departments and  SOCOM, DOD proposed a $78 billion reduction in its overall budget  plan over a 5-year time period, covering fiscal years 2012 through  2016, which reflected a 2.6 percent reduction from DOD’s fiscal year  2011 budget submission over the same time period. Of this amount,  $54 billion reflected projected savings identified from a health care  policy assessment, government-wide civilian pay freeze, and other  specific areas identified by the Secretary where immediate action  could be taken department-wide. The remaining $24 billion reflected  revised economic assumptions, projected savings from restructuring  the Joint Strike Fighter weapon program, and projected savings from  reducing the size of the Army and Marine Corps.
Information accompanying DOD’s fiscal year 2012 budget request  catalogued the $100 billion in savings from the military departments and  SOCOM under the following four categories:  reorganizations, such as restructuring headquarters management and  eliminating unneeded task forces; better business practices, such as reducing energy consumption; program reductions and terminations, such as terminating weapon  system programs; and  reductions in lower priority programs, such as shifting funding  requests from military construction projects to base operations.
Table 1 shows the specific amounts of projected savings reported for  each category and military department.
Among other things, the  the budget account from which each savings identified will be derived;  the number of military personnel and full-time civilian employees of  the federal government affected by such savings;  the estimated reductions in the number and funding of contractor  personnel caused by such savings; a specific description of activities or services that will be affected by  such savings, including the locations of such activities or services;  and  certain information regarding each reinvestment planned to be funded  with efficiency initiative savings.
In June 2012, DOD submitted its report to the congressional defense  committees and provided some information on the categories above.  Among other things, at that time, DOD reported that it was on track to  meet estimated savings targets for all of its efficiency initiatives. However,  the report did not include a comprehensive analysis of reinvestments  because, according to DOD, many areas in which reinvestments had  occurred due to the efficiency initiatives included in the fiscal year 2012  President’s Budget request had been offset by major force structure  changes and other reductions in its fiscal year 2013 budget request. In  briefings to the Comptroller delivered in February 2013 and March 2013,  the military departments and SOCOM reported that they remain, with a  few exceptions, on track to meet original savings estimates associated  with their individual efficiency initiatives. At the time of our review, DOD  had not compiled DOD-level summary information on its progress in  achieving its original savings estimate of $100 billion. DOD officials cited  various reasons why compiling and reporting on this information may not  be feasible. For example, they noted that the need to apply spending  reductions in response to sequestration affected funding levels for many programs, including areas targeted for efficiency initiatives. As a result,  DOD had to adjust plans for executing programs as well as for  implementing initiatives, such as adjusting the scope of initiatives or the  timing of actual or planned actions for implementation. Because of the  variability in its programs and funding amounts, DOD officials stated that,  at a certain point, it becomes difficult to isolate whether savings were  achieved solely due to implementing initiatives rather than a combination  of factors.
We have previously reported on opportunities for DOD to improve  tracking and reporting on cost savings and efficiencies. For example, in  March 2012, we reported that DOD took steps to examine its  headquarters resources for potential efficiencies, but that it faced an  underlying challenge of not having complete and reliable headquarters  information available to make related assessments and decisions. To  improve DOD’s ability to identify how many headquarters personnel it  has, we recommended that the Secretary of Defense direct the Director of  Administration and Management, in consultation with the Under Secretary  of Defense for Personnel and Readiness, to revise its DOD Instruction  5100.73, Major DOD Headquarters Activities, to, among other things,  include all major DOD headquarters activity organizations. DOD partially  concurred with our recommendation and commented that the  shortcomings in the instruction have limited impact on the management of  the department. In July 2012, we reported that, as part of one of its  efficiency initiatives, the Air Force estimated it could save about $1.7  billion in its training program by reducing live flying hours and taking other  steps, such as increasing the use of virtual training, but lacked a  methodology for determining the costs of virtual training and therefore, did  not consider these costs in its estimate.visibility over the costs related to virtual training, we recommended that  the Secretary of Defense direct the Secretary of the Air Force to develop  a methodology for collecting and tracking cost data for virtual training and  use this cost data to help inform future decisions regarding the mix of live  and virtual training. DOD concurred with our recommendation and   To improve decision makers’  identified actions being taken to enhance its ability to capture costs  related to virtual training.
Additionally, in December 2012, we reported that DOD had developed an  approach for the military departments and SOCOM to follow in tracking  and reporting on the status of efficiency initiatives; however, DOD’s  approach had some limitations that resulted in incomplete reporting which  could limit the visibility of senior leaders in monitoring progress toward  achieving programmatic and financial goals. Specifically, the offices of the  Comptroller and DCMO had provided general direction through emails,  briefings, and training, and, according to officials, had given the military  departments and SOCOM flexibility to report on the efficiency initiatives  that they felt were most important. In practice, the Army, Air Force, and  SOCOM had reported on all of their efficiency initiatives, while the Navy  reported on a subset of its initiatives based on what it deemed to be at  medium or high risk of experiencing implementation issues or adversely  affecting the Navy’s ability to carry out its mission. With respect to  realized savings, we reported that the military departments and SOCOM  told us they were on track to realize estimated savings, but found some  instances where certain costs were not considered. For example, for its  initiative to reduce fleet shore command personnel from U.S. Pacific Fleet  and U.S. Fleet Forces Command, the Navy did not account for potential  increases in relocation costs for moving personnel to other areas within  the Navy. We found that the military departments and SOCOM were not  reporting consistent information or complete cost information because  they had not received written guidance with standardized definitions and  methodologies. Rather, the direction provided by the offices of the  Comptroller and DCMO did not specify whether all of the costs associated  with implementing an efficiency initiative, including costs not initially  identified, should be included. To ensure more complete and consistent  reporting, we recommended that DOD develop guidance with  standardized definitions and methodologies for the military departments  and SOCOM to use in reporting. Further, we recommended that guidance  should define reporting requirements for such things as the specific types  of costs associated with implementing the initiatives, including  implementation costs that were not initially identified in calculations of net  savings. DOD agreed with the spirit and intent of our recommendation  and indicated it planned to issue additional formal guidance in the  February 2013 timeframe. The status of DOD’s implementation of this  recommendation is discussed in more detail later in this report.
Since initiating its initial round of initiatives for fiscal year 2012, DOD has  continued to identify and implement efficiency initiatives. Specifically, in  information accompanying its fiscal years 2013 and 2014 budget  requests, DOD identified additional efficiency initiatives, referred to as  More Disciplined Use of Resources (MDUR) initiatives. These initiatives  are expected to generate $60 billion in savings for the period of fiscal  years 2013 through 2017 and an additional $34 billion for the period of  fiscal years 2014 through 2018. While savings generated by the  Secretary’s fiscal year 2012 efficiency initiatives were to be reinvested,  savings from the MDUR initiatives were intended to help the department  meet reductions to its budget, and therefore will not be reinvested. More  recently, on July 31, 2013, as part of DOD’s Strategic Choices and  Management Review, the Secretary announced his plan to implement an  additional round of efficiency initiatives in fiscal year 2015. According to  the Secretary, a tenet of the review was the need to maximize savings  from reducing DOD’s overhead, administrative costs, and other  institutional expenses. These initiatives would include management  reforms, coupled with consolidations, personnel cuts, and spending  reductions that would reduce DOD’s overhead and operating costs by  some $10 billion over the next 5 years and almost $40 billion over the  next decade.
section_title: DOD Has Taken Steps to Further Refine Its Approach to Track and Report on Its Efficiency Initiatives
paragraphs: DOD has taken steps to further refine its approach to its tracking and  reporting on the implementation of its efficiency initiatives. Specifically,  DOD issued written guidance that standardizes and expands the type of  information on efficiency initiatives that the military departments and  SOCOM are expected to report, which may improve visibility on the  progress and risks in implementation for DOD decision makers.   Moreover, in commenting on a draft of this report, DOD stated that it will  cease to track initiatives once they have been implemented and will select  for detailed tracking only those initiatives where this information will help it  manage more effectively.
Following our December 2012 report, the Comptroller issued written  guidance in February 2013 establishing a standardized format for  reporting on the fiscal year 2012 efficiency initiatives as well as the fiscal  year 2013 MDUR initiatives. According to DOD officials, this guidance is  also applicable to initiatives identified in fiscal year 2014 and any future  initiatives. In contrast to the way they reported before, the military  departments and SOCOM were now expected to report consistently and  provide the status of their efficiency initiatives, including summary  information related to (1) whether original net savings projections across  the Future Years Defense Program are being met, (2) risks to  program(s), mission(s), or resources associated with the efficiency  initiative (characterized as “low”, “medium”, or “high” risk), and (3) any  risks to “milestones” or the implementation status of the efficiency  initiative (e.g., characterized as “on track,” “off track but can meet major  milestones,” or “off track and cannot meet major milestones”). Only in  instances where the military departments and SOCOM identified  programs that were not achieving original net savings estimates or where  program or milestone risk had been identified, the guidance requires  further detail, including how implementation would be achieved. Further,  all of the information was to be reported in a manner that mirrored the  descriptions contained in DOD’s fiscal year 2012 budget request  justification book for the efficiency initiatives, whereby some of the  efficiency initiatives were collapsed into broader groups of initiatives  referred to by descriptive titles. In February and March 2013, using the  new February 2013 guidance, the military departments and SOCOM  completed the first round of semi-annual reporting on the fiscal year 2012  efficiency initiatives and fiscal year 2013 MDUR initiatives.
In reviewing the military departments’ and SOCOM’s February 2013 and  March 2013 reports, we observed that, consistent with the  aforementioned February 2013 guidance, the military departments and  SOCOM reported details associated with only those efficiency initiatives  that were not achieving original net savings estimates or where program  or milestone risk had been identified. As a result, detailed information on  the full range of efficiency initiatives and related programs was not  included in their reports. For example, the Air Force has as many as 10  individual initiatives that comprise its acquisition management initiative.  Absent a requirement in the February 2013 guidance to report on each of  those underlying initiatives, DOD decision makers would only receive  information on the overall acquisition management initiative. Moreover, as  a result of the reporting direction, DOD decision makers would receive  detailed information on the overall acquisition management initiative only  if the initiative is not meeting original savings estimates, or where  program or milestone risk had been identified. Prior to the February 2013  guidance, some departments and SOCOM had previously chosen to  report on all their initiatives. In reviewing the reports developed by the  military departments and SOCOM in February 2013 and March 2013, we  observed that information on all initiatives was now unavailable to DOD  decision makers, thus hindering their ability to assess implementation  progress across the full range of efficiencies. We discussed with DOD  Comptroller officials whether reporting on only those efficiency initiatives  not achieving their original estimates or facing risk had provided the  Comptroller with sufficient details to oversee all of the initiatives.  Comptroller officials agreed that reporting on each of the individual  efficiency initiatives would improve DOD decision makers’ visibility and  therefore provide information needed for their oversight. They also noted  that it would facilitate DOD’s ability to address any future congressional  reporting requirements. As a result, the Comptroller’s office subsequently  issued guidance in October 2013 that, according to these officials,  superseded the February 2013 guidance and expanded the amount of  information to be reported. Specifically, this guidance directed the military  departments and SOCOM to submit further detail for all efficiency  initiatives, rather than merely those not achieving the original estimates or  at risk. Beginning in October 2013, the military departments and SOCOM  began submitting reports that included this broader set of information.
While obtaining this broader set of information, DOD stated in its written  comments on a draft of this report, provided on January 6, 2014, that it  will narrow the scope of efficiency initiatives that will be tracked due to the  period of constrained resources it is experiencing. DOD stated that it will  cease tracking initiatives once they have been implemented, and will  select for detailed tracking only those initiatives where this information will  help it manage more effectively. In clarifying its written comments, DOD  officials stated that all of its efficiency initiatives, except those  implemented or which strictly call for terminations of programs, such as  weapons systems, will be selected for detailed tracking. While the  October 2013 guidance, which requires the military departments and  SOCOM to report more detailed information on the full range of ongoing  efficiency initiatives, does not specify that initiatives that strictly call for  program terminations should no longer be tracked, we believe it is  reasonable for DOD to cease the tracking of initiatives that strictly call for  program terminations.
We note that, in issuing additional guidance on its tracking and reporting  on efficiency initiatives, DOD did not include any direction as to the  specific types of costs that the military departments and SOCOM should  consider in determining realized savings associated with implementation,  such as costs that were not initially identified in calculations of net  savings, as we had recommended in our December 2012 report.  According to a Comptroller official, DOD has various guidance on  developing cost estimates that the military departments and SOCOM can  use in determining savings associated with the implementation of their  efficiency initiatives. We reviewed the documents and discussed with the  efficiency initiative focal points how, or if, this guidance was applied in  developing their cost estimates. Some of the program managers with  whom we spoke confirmed that while they were aware of existing  guidance on developing cost estimates, they had not been instructed to  use this guidance to determine specific types of costs that should be  considered in calculating net savings. As a result, we continue to believe  that our prior recommendation in our December 2012 report has merit  and should be implemented.
section_title: Military Departments and SOCOM Vary in the Extent and Nature of Their Efforts to Evaluate the Impact of Their Efficiency Initiatives
paragraphs: The military departments and SOCOM have taken steps to evaluate  some of their efficiency initiatives, such as establishing performance  measures and collecting performance data. However, these efforts have  largely occurred on an ad hoc basis and vary by efficiency initiative  because DOD has not established a requirement for performing such  evaluations. As a result, DOD lacks a systematic basis for evaluating the  impact of its efficiency initiatives on improving program efficiency or  effectiveness.
In setting forth the initial efficiency initiatives, the Secretary of Defense  intended for DOD to improve the effectiveness and efficiency of its  programs and activities. The Secretary also directed that any efficiency  initiative must be specific, actionable, and measurable. Our prior review of  federal agencies’ efficiency efforts concluded that an improvement in  efficiency need not only involve a reduction in costs, but also can be  achieved by maintaining federal government services or outcomes with  fewer resources (such as time or money), or improving or increasing the  quality or quantity of services while maintaining (or reducing) resources.  In addition, we and other agencies, such as the Office of Management  and Budget (OMB), have documented the need to develop performance  measures for evaluating progress toward achieving desired outcomes.  For example, as we have previously concluded, performance measures  should be measurable, outcome-oriented, and actively tracked and  reported. As our prior work has shown, leading organizations that employ  result-oriented management use performance information as a basis for  decision making and have found this approach improves program  results.
As previously discussed, the Comptroller’s October 2013 guidance  provides direction on how the military departments and SOCOM are to  approach reporting on the status of their efficiency initiatives, but does not  require them to develop approaches for evaluating the impact of initiatives  on achieving desired outcomes. In practice, we found that the military  departments and SOCOM varied in the extent to which they evaluated  individual efficiency initiatives, including whether they had established  measures or indicators to gauge the impact on program efficiency or  effectiveness beyond savings. The following paragraphs provide  examples of the services’ and SOCOM’s efforts to evaluate certain  efficiency initiatives.
Air Force’s Facility Sustainment Initiative: This initiative is intended to  reduce infrastructure maintenance costs by a total of $1.4 billion  during the period of fiscal years 2012 through 2016 by performing  preventative maintenance before critical failures occur. The Air Force  uses a model to predict and prioritize infrastructure most at risk for  critical failures and then focuses preventive maintenance efforts on  such infrastructure. Furthermore, the Air Force has established  measures to track the amount of hours spent performing preventive  and corrective maintenance over the course of the initiative to  determine whether this effort achieved the intended outcome, which  was to reduce the amount of more costly corrective maintenance  performed. We have previously concluded that deferring facility  sustainment can lead to shortened facility service lives and increased  future costs for recapitalization.
SOCOM’s Information Technology Services Efficiency Initiative:  SOCOM established a new approach for its information technology  services that is intended to reduce costs by a total of $394 million  during the period of fiscal years 2012 through 2016. According to  SOCOM officials, the new approach involved a contract framework for  information technology services that reduces costs by awarding funds  directly to the organizations that provide the services on a competitive  basis, rather than through an intermediary that selects the  organizations that provide the information technology services. The  approach also adopts other best practices for procurement, such as  providing performance-based incentives. As part of this initiative,  SOCOM established multiple individual measures to assess  contractor performance, such as answering a help desk call within a  set amount of time or tracking trends on resolving information  technology issues such as user access, but does not have measures  in place to evaluate how the overall impact of the initiative affects the  delivery of information technology services relative to the previous  approach. SOCOM officials explained that because the  implementation of its Information Technology Services Efficiency  Contract occurred prior to it being identified as an efficiency initiative,  methods of evaluating its effectiveness or efficiency, other than cost,  were not established.
Navy’s Total Ownership Cost Initiative: This initiative seeks to achieve  efficiencies through life cycle management of the Navy’s ships and  encompasses multiple underlying initiatives, such as the Navy’s  Revised Virginia Class Drawings Efficiency Initiative. This initiative is  intended to reduce costs of $30.3 million during the period of fiscal  years 2012 through 2016 by moving away from reliance on paper  documents toward an electronic system that allows multiple users to  make revisions and access up-to-date documents. The Navy has not  yet identified measures to evaluate how increasing the use of the  electronic system to process technical documents will maintain or  improve work processes. Navy officials indicated the same is true for  other fiscal year 2012 initiatives that make up the Navy’s Total  Ownership Cost initiative and commented that the focus was on  tracking savings and not on developing efficiency measures to assess  whether its initiatives, once implemented, improve the effectiveness or  efficiency of these programs.
Army and Air Force Data Center Consolidation Efficiency Initiatives:  These initiatives are part of the larger Federal Data Center  Consolidation Initiative, directed by OMB, that seeks to consolidate  information technology infrastructure and activities to save energy  costs, among other goals, and are expected to reduce the Army and  Air Force’s costs by $490 million and $180 million, respectively, during  fiscal years 2012 through 2016. Both the Army and Air Force have  taken steps to implement these initiatives. According to Air Force  officials, they had begun to establish measures that could be used to  evaluate the impact of these initiatives, but faced some challenges  due to changing guidance. For example, officials discussed that OMB  guidance issued after the initiatives were underway expanded the  definition of a data center and effectively increased the scope of the  military departments’ consolidation effort after these initiatives were  submitted in the fiscal year 2012 budget request. This resulted in the  reconfiguration of planning and implementation schedules. Air Force  officials also stated that they had begun to develop performance  measures to assess impact when a new DOD effort to establish a  secure, joint information environment was put in place. Therefore,  the Air Force had to adjust their implementation plans and postpone  the development of their measures to ensure actions taken on this  initiative conformed to the new DOD requirement.
While the focus of DOD’s effort was to quickly identify funds that could be  reinvested into other higher priority programs, military department and  SOCOM officials explained that because the initial effort to identify  efficiency initiatives occurred late at the end of the cycle they used to  build their fiscal year 2012 budget submission, their effort focused on  tracking savings targets and not on developing measures to evaluate  impact. In the subsequent budget cycles that included the MDUR  initiatives, the focus remained on identifying areas for reductions in  spending. DOD officials agreed that additional measures could be useful  to evaluate impacts—beyond savings—of their efficiency initiatives. Our  prior work concluded that such measures can assist managers in  determining whether desired outcomes were being achieved or if  adjustments were needed, such as in the scope of the initiative or to the  nature or timing of implementation actions. Without a systematic way to  evaluate the impact of its efficiency initiatives, DOD is limited in its ability  to assess whether the efficiency initiatives have improved the  effectiveness or efficiency of its programs and activities.
section_title: Conclusions
paragraphs: Over the past few years, in light of mounting fiscal pressures, DOD has  continued to identify and implement efficiency initiatives with certain goals  in mind, including achieving cost savings and seeking opportunities to  enhance the efficiency or effectiveness of its programs and activities.  DOD’s recent efforts to refine its approach for tracking and reporting on  its current efficiency initiatives has the potential for providing greater  oversight to decision makers on progress of the military departments and  SOCOM on the status of their implementation efforts. However, its efforts  to date do not sufficiently ensure that leaders have the information they  need to fully assess the impact these initiatives are having on DOD’s  programs and activities. Having a systematic way to evaluate the impact  of its efficiency initiatives beyond cost savings could provide DOD the  ability to determine whether or not its initiatives are improving the  efficiency and effectiveness of its programs and activities while also  achieving savings. Such information could also inform DOD as to whether  actions are needed to make adjustments to the scope of any given  initiative and related programmatic actions necessary for implementation.
section_title: Recommendation for Executive Action
paragraphs: To enhance DOD’s ability to determine whether its efficiency initiatives  are having the desired effect of improving efficiency and effectiveness, we  recommend that the Secretary of Defense require the military  departments and SOCOM to develop approaches for evaluating the  impact of their efficiency initiatives, such as establishing performance  measures or other indicators, collecting related performance information,  and using this information to measure progress in achieving intended  outcomes associated with their initiatives until implemented.
section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of this report, DOD concurred with our  recommendation and provided additional comment. Specifically, DOD  concurred with having the military departments and SOCOM develop  performance measures and other indicators for evaluating the impact of  its efficiency initiatives, and commented that it has decided to cease  tracking initiatives once they have been implemented. DOD also provided  technical comments, which were incorporated as appropriate. The full text  of DOD’s comments is reprinted in appendix II.
In its overall comments, DOD stated it intends to continue to refine its  procedures, guidance, and oversight in order to achieve its goal of  identifying and implementing efficiencies. DOD also stated that during this  period of constrained resources, it must avoid creating a costly and  redundant oversight process. To that end, DOD stated that it will cease  tracking initiatives once they have been implemented, and will select for  detailed tracking only those initiatives where this information will help it  manage more effectively. We have modified the report to reflect DOD’s  decision.
In clarifying its written comments, DOD officials stated that DOD will  select for detailed tracking all of its efficiency initiatives except those  implemented or which strictly call for terminations of programs, such as  weapons systems. While the October 2013 guidance does not specify  that initiatives that strictly call for program terminations should no longer  be tracked, in a resource-constrained environment, we believe it is  reasonable for DOD to do so. We also expect that DOD will clarify this  revised approach in any future guidance to the military departments and  SOCOM.
In addition, we have modified the recommendation to clarify its intent that  DOD should develop an approach for evaluating the impact of its  efficiency initiatives until those initiatives have been implemented.
We are sending copies of this report to the Secretary of Defense and  appropriate congressional committees. In addition, the report is available  at no charge on the GAO website at http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-5257 or merrittz@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. Key contributors to this report are listed in appendix  III.
section_title: Appendix I: Scope and Methodology
paragraphs: To determine the progress DOD has made in adjusting its approach to  tracking and reporting on the implementation of its efficiency initiatives  since we last reported in December 2012 and to assess the extent to  which DOD is evaluating the impact of its efficiency initiatives on DOD  programs and activities, we reviewed guidance and documentation issued  at the department-wide level as well as within the military departments  and SOCOM. We also interviewed officials from the offices of the  Comptroller and DCMO, the military departments, and SOCOM who are  involved in monitoring the implementation of its efficiency initiatives to  discuss their approach to tracking and reporting on the initiatives.  Specifically, we obtained available information from each of the military  departments and SOCOM, including briefings prepared for senior DOD  officials, on the current status of initiatives, how original estimates of  savings compared with savings realized to date, and any program or  timeline risks associated with implementing the efficiency initiatives.  Additionally, we reviewed existing guidance to identify any requirements  for evaluation of the efficiency initiatives. We then analyzed information  provided by each of the military departments and SOCOM as well as  interviewed officials from each of the military services and SOCOM  serving as focal points for the efficiency initiatives to determine the  expected outcome or impact for individual initiatives, and what steps they  have taken to evaluate the impact of efficiency initiatives.
We conducted this performance audit from May 2013 through January  2014 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
section_title: Appendix II: Comments from the Department of Defense
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Tina Won Sherman, Assistant  Director; Grace Coleman; Susan Langley; Ricardo A. Marquez; Sharon L.  Pickup; Mike Silver; Michael Shaughnessy; Susan Tindall; and Sarah  Veale made key contributions to this report.