Data Center Optimization: Agencies Need to Complete Plans to Address Inconsistencies in Reported Savings
Released: 2017-05-18T00:00:00

Letter

Conclusions

Continuing to build on progress made under FDCCI, agencies have reported significant data center closures—totaling more than 4,300 through August 2016—and current plans identify more than 1,200 additional centers planned for closure through fiscal year 2019. If executed as planned, such closures will help agencies continue to increase their operational efficiency and achieve savings. Agencies have also reported significant consolidation and optimization cost savings and avoidances. However, agencies’ total planned cost savings are $3.3 billion less than what we reported in 2015. In the absence of planned savings that remain consistent and fully reported, total planned savings are likely to continue to be understated.

OMB’s recent guidance on data center optimization was a positive step forward in providing agencies with the detailed guidance to establish their required strategic plans to optimize their data centers and achieve cost savings. However, not all agencies have submitted their plans and only seven of the plans submitted addressed all the key elements required by FITARA and OMB. Moreover, significant inconsistencies in agencies’ reported savings raise further questions regarding the accuracy of agency reporting and OMB’s oversight of this initiative. Addressing these inconsistencies and expeditiously completing these plans is increasingly important in order to meet the federal requirements. Until OMB improves its oversight of the information in agencies’ strategic plans, and agencies complete the missing elements, they may be challenged in fully implementing the data center consolidation and optimization provisions of FITARA. Further, until OMB improves it oversight of agencies’ reporting of cost savings and agencies address inconsistencies in their reporting, savings will likely continue to be underreported.

Recommendations for Executive Action

To better ensure that federal data center optimization efforts improve governmental efficiency and achieve cost savings, we are making two recommendations to OMB. Specifically, we recommend that the Director of OMB direct the Federal CIO to provide the necessary oversight to ensure that each agency completes their DCOI strategic plan in accordance with OMB’s guidance implementing FITARA and ensure that agency reporting of achieved data center consolidation and optimization cost savings and avoidances is consistent across all reporting mechanisms, including quarterly data submissions and agency DCOI strategic plans.

We also recommend that the following 17 agencies (the Secretaries of the Departments of Commerce, Defense, Energy, Health and Human Services, Interior, Labor, State, Transportation, Treasury, and Veterans Affairs; the Attorney General; and the Administrators of the Environmental Protection Agency, National Aeronautics and Space Administration, Small Business Administration, and U.S. Agency for International Development; the Chairman of the Nuclear Regulatory Commission; and the Commissioner of the Social Security Administration) each take action to complete the missing elements in their respective DCOI strategic plan, including addressing any identified challenges, and submit their completed strategic plan to OMB.

Finally, we recommend that the following 11 agencies (the Secretaries of the Departments of Commerce, Education, Health and Human Services, Interior, Labor, State, Transportation, and Treasury; the Administrators of the Environmental Protection Agency, General Services Administration, and the U.S. Agency for International Development) also each take action to ensure that the amounts of achieved data center cost savings and avoidances are consistent across all reporting mechanisms, including the quarterly data submissions and DCOI strategic plans.

Agency Comments and Our Evaluation

We received comments on a draft of this report from OMB and from the 24 agencies that we reviewed. Of these, 12 stated they agreed with our recommendations, 2 (the Department of Housing and Urban Development and the Nuclear Regulatory Commission) did not agree, and 11 (including OMB) did not state whether they agreed or disagreed. Multiple agencies also provided technical comments, which we have incorporated as appropriate. The following discusses each agency’s comments.

In an e-mail received on April 5, 2017, a staff member from OMB’s Office of General Counsel did not agree or disagree with our recommendation, and offered several comments on our findings. Specifically, the staff member stated that OMB was unable to replicate our totals of the number of data centers closed and remaining. The staff member also noted that we did not include agency cloud service providers or data centers with no closing date in our totals, whereas OMB includes such instances in its counts. In conducting our analysis, we did not count cloud service providers because OMB’s guidance specifically states that such providers are not to be considered as data centers. In regard to data centers identified as planned for closure, but without a closing date, we did not count such facilities as closures because the lack of a date suggested only a tentative plan, rather than a firm commitment.

The staff member further stated that OMB could not replicate our cost savings estimates and noted that while OMB relied solely on information reported by agencies to the IT Dashboard, we also had access to contextual information about the data, provided to us by the agencies. The staff member asserted that OMB did not have access to that same information. However, the contextual information referenced by the staff member is only amplifying information we obtained from the agencies through the normal course of our audit work, as we interviewed agency officials in order to verify the information reported by the agencies to OMB and to verify the results of our analyses. On multiple occasions, our work has pointed out the differences in what agencies are reporting to OMB and what they are reporting to us, and we have made multiple recommendations for OMB and agencies to address these discrepancies. For this reason, we requested and received agencies’ validation of the results of our analysis of agency closures and cost savings on multiple occasions. As such, we stand by the results of our analysis and the data center closures and cost savings estimates that we are reporting.

Additionally, the staff member stated that grouping together in one category the agencies that had not fully completed all the required elements of their DCOI strategic plans ignored significant quality differences between the plans, as well as shared problems with each of them. Our report currently summarizes that seven agencies addressed all required plan elements, but that 17 other agencies either partially met or did not meet the requirements. Our report also provides a graphic that illustrates the number of agencies that met, partially met, or did not meet each requirement. Further, the briefing slides that we are transmitting with this report (found in appendix I) detail the extent to which individual agencies completed their strategic plans and also discuss reasons why agencies were unable to complete their plans, including shared challenges. As such, we maintain that the statement in our report appropriately summarizes the findings detailed in our briefing.

In an e-mail received on April 3, 2017, a senior advisor in the Department of Agriculture’s Office of the CIO stated that the CIO concurred with our recommendation that the department had taken action to complete the missing elements in its strategic plan, including addressing any identified challenges, and submitted the completed strategic plan to OMB. The advisor noted that the department had updated its DCOI strategic plan to include fiscal year 2016 achievement information and posted its revised plan. In addition, the advisor said that the department had posted its CIO statement asserting that the agency has complied with all reporting requirements in OMB’s memorandum and the data center requirements of FITARA.

Our analysis of the department’s updated DCOI strategic plan concluded that the plan included the missing elements we identified during our review. We also confirmed that the department’s CIO statement was available publicly on the department’s website. As a result, we have removed this recommendation from our report. In addition, the department provided technical comments, which we have incorporated as appropriate.

In written comments, the Department of Commerce agreed with our recommendations and described planned actions to address them. Specifically, the department stated that it will continue to aggregate its data center inventory and update its DCOI strategic plan by OMB’s April 17, 2017, submission deadline. The department also stated that it will, through the integrated data collection process, continue to collect and report all initiatives resulting in cost savings and avoidances to ensure IT savings are being captured and realized. The Department of Commerce’s comments are reprinted in appendix II.

In an e-mail received on March 27, 2017, a Department of Defense CIO audit liaison stated that the department had no comment on the draft report, and did not state whether the department agreed or disagreed with the recommendations. The department provided technical comments, which we have incorporated as appropriate.

In written comments, the Department of Education did not agree or disagree with our recommendations but acknowledged that the department had not included historical cost savings and avoidance information in its DCOI strategic plan, as required. The department attributed the omission to its interpretation of DCOI strategic plan guidance, and provided a life cycle cost estimate, rather than the required historical cost savings through 2015. The department stated that it has subsequently corrected its strategic plan, and it provided us with a copy of the plan.

Our draft report provided to the department for comment included a recommendation that the department take action to complete the missing elements in its strategic plan, including addressing any identified challenges, and submit the completed strategic plan to OMB. Our analysis of the department’s updated DCOI strategic plan states that it includes the missing elements we identified during our review. We also confirmed that the department’s CIO statement was available publicly on the department’s website. As a result, we have removed this recommendation from our report. The Department of Education’s comments are reprinted in appendix III.

In written comments, the Department of Energy concurred with our recommendation that it take action to complete the missing elements in its DCOI strategic plan, including addressing any identified challenges, and submit the completed strategic plan to OMB. The department stated that it had submitted its fiscal year 2016 DCOI strategic plan to OMB in February 2017 and posted a copy on an agency web site. The department further said that in the plan, it reported meeting or exceeding all fiscal year 2016 goals and established performance targets for fiscal years 2017 and 2018, and that the plan included all past and present cost savings and avoidances achieved. Further, the department noted that it expects to complete a comprehensive inventory of its data centers by the release date for the fiscal year 2017 DCOI strategic plan in April 2017.

Our analysis of the department’s updated strategic plan found that it includes most of the missing elements we identified during our review. For example, it includes target optimization, closure, and savings metrics for fiscal years 2016 through 2018 and achieved data for those metrics for 2016. However, the plan does not include the costs of closures and optimization, two key components of OMB’s required cost savings metric. In the updated plan, the department states that it intends to include such information in the next plan update. Because the department’s updated DCOI strategic plan did not include the costs of closures and optimization, we maintain that the recommendation is still valid. The Department of Energy’s comments are reprinted in appendix IV.

In an e-mail received on April 3, 2017, a Department of Health and Human Services audit liaison stated that the department had no comments on the draft report, and did not state whether the department agreed or disagreed with our recommendations.

In written comments, the Department of Homeland Security concurred with our recommendation that it take action to complete the missing elements in its strategic plan, including addressing any identified challenges, and submit the completed plan to OMB. The department stated that, in March 2017, its Office of the Chief Information Officer finalized its DCOI strategic plan and uploaded it to an agency web page, as directed by OMB. The department further noted that the plan contained all required elements and discussed challenges identified by the department. Finally, the department said that its DCOI metrics and its CIO certification had been uploaded to the web page, as well.

Our analysis of the department's updated DCOI strategic plan found that it includes the missing elements we identified during our review. We also found that the department's CIO statement was available publicly on the department's website. As a result, we have removed this recommendation from our report. In addition, the department provided technical comments, which we have incorporated as appropriate. The Department of Homeland Security's comments are reprinted in appendix V.

In written comments, the Department of Housing and Urban Development disagreed with our recommendation to take action to complete the missing elements in its DCOI strategic plan, including addressing any identified challenges, and to submit its completed strategic plan to OMB. Specifically, the department requested that it be removed from the list of agencies that had not completed elements in their strategic plan. The department noted that its data center consolidation efforts were complete and associated cost savings realized at least 10 years prior. Consequently, the department had completed the estimated savings section of its plan as "not applicable," an option provided by OMB.

After further review of the department's DCOI strategic plan, we agree with the department's assertion that it provided a complete plan. As a result, we have removed this recommendation from our report.

The department also pointed to multiple locations in the report where it stated we had inaccurately assessed the state of its data center consolidation efforts and associated savings, and the completeness of the department's DCOI strategic plan. However, all of the specific instances cited by the department are found in appendix I of this report, which is a briefing that we delivered to Congress on December 19, 2016. Before we delivered our briefing, the department was provided two separate opportunities to comment on the facts and findings contained in our briefing. Specifically, on November 21, 2016, we provided the department with a copy of the briefing for factual correction and on December 6, 2016, we provided a copy of the briefing for comment on the conclusions and recommendations. In response to our first request, a departmental liaison replied on November 30, 2016, that the department had no factual corrections to the briefing. The department did not respond to our second request, despite several attempts that we made to obtain its comments.

The department did not raise these concerns over the factual statements in our briefing until its March 29, 2017, written comments on the draft copy of this report. As this report serves to publish our briefing that was delivered to Congress in December 2016, we have not revised the contents of the briefing that is re-printed in appendix I. However, as noted, we have re-evaluated our characterization of the department’s strategic plan that is found in the body of this report, have updated our assessment of the completion of the plan, and removed our recommendation. The Department of Housing and Urban Development’s comments are reprinted in appendix VI.

In written comments, the Department of the Interior agreed with our recommendations and stated that it would take action to complete the missing items in its DCOI strategic plan, including addressing any identified challenges, and would submit the completed plan to OMB. The department also said it would ensure that the amounts of achieved data center cost savings and avoidances are consistent across all reporting mechanisms, including the quarterly data submissions and DCOI strategic plans. The Department of the Interior’s comments are reprinted in appendix VII.

In an e-mail received March 28, 2017, an audit liaison specialist in the Justice Management Division of the Department of Justice stated that the department had no formal comments on the draft report. The liaison also did not state whether the department agreed or disagreed with our recommendation that it take action to complete the missing elements in its strategic plan, including addressing any identified challenges, and submit the completed strategic plan to OMB.

However, the liaison did provide us a copy of an updated DCOI strategic plan and a copy of the department's CIO statement.

Our analysis of the department's updated DCOI strategic plan determined that it includes most of the missing elements we identified during our review. For example, it includes the achieved data values for optimization metric performance, data center closures, and data center consolidation and optimization-related cost savings for fiscal year 2016. We also found that the department's CIO statement was available publicly on the department's website.

Nevertheless, while the updated DCOI plan provides a description of costs for data center consolidations and optimization in fiscal year 2016, it does not provide dollar amounts for the data center investments. Further, while we recognize that the updated plan addresses the majority of the requirements, the department did not include the dollar amounts for its fiscal year 2016 investments in data center consolidation and optimization, as required by OMB's guidance on the implementation of FITARA. Thus, we maintain that our recommendation is still valid. The department also provided technical comments, which we incorporated as appropriate.

In an e-mail received on April 3, 2017, a Department of Labor audit liaison stated that the department’s technical staff had no comments on the draft report and did not state whether the department agreed or disagreed with our recommendations.

In written comments, the Department of State agreed with our recommendations and described planned actions to address them. Specifically, the department described plans to acquire, subject to funding availability, automated monitoring tools for its enterprise data centers. It also described plans to engage OMB to rebaseline the closure target for its non-tiered data centers located outside the United States, based on the department’s mission needs. In addition, the department noted that it is in the process of identifying the number of server rooms in the United States that meet the DCOI definition of a data center. The department also reaffirmed its commitment to include updated data center closures and cost savings information in its DCOI strategic plan. The Department of State’s comments are reprinted in appendix VIII.

In an e-mail received on March 6, 2017, an audit liaison in the Department of Transportation’s Audit Relations and Program Improvement office stated that the department concurred with our recommendations.

In an e-mail received on March 31, 2017, an audit liaison in the Department of the Treasury’s Office of the CIO did not agree or disagree with the recommendations, but commented on our findings. Specifically, the comments noted that the differences we identified in Treasury’s reported savings were due to a change in calculation methodology. The comments further noted that the department’s February 2017 quarterly report to OMB included both the historical savings identified under the old methodology and the current DCOI savings initiative, to better correlate with savings figures previously reported to us and with savings figures reported in the department’s DCOI strategic plan. We agree that, while the gap in Treasury’s reported savings to OMB, as compared to savings reported to us, has narrowed substantially, there is still a shortfall between what the department recently reported and what was previously reported to us.

In written comments, the Department of Veterans Affairs agreed with our recommendation that it take action to complete the missing elements in its DCOI strategic plan, including addressing any identified challenges, and submit the completed strategic plan to OMB. The department further stated that it had completed and resubmitted its DCOI strategic plan to OMB in December 2016 and that it had posted the plan on an agency web page.

Our analysis of the department's updated plan determined that the updated plan includes several of the missing elements we identified during our review. For example, it includes the achieved data values for optimization metric performance, data center closures, and data center consolidation and optimization-related cost savings for fiscal year 2016.

However, while the updated DCOI plan provides a description of costs for data center consolidations and optimization in fiscal year 2016, it does not provide dollar amounts for the data center investments. While we found that the department's updated plan addresses most of the requirements, the plan does not include the dollar amounts for the department's fiscal year 2016 investments in data center consolidation and optimization, as required by OMB's guidance on the implementation of FITARA. Further, we also did not find the department's CIO statement available publicly on the department's website. For these reasons, we maintain that our recommendation is still valid. The Department of Veterans Affairs' comments are reprinted in appendix IX.

In written comments, the Environmental Protection Agency did not agree or disagree with our recommendations, but described actions planned to address them. Specifically, the agency stated that it will update its DCOI strategic plan to include elements not reflected in the 2016 submission and will complete the plan to the extent feasible. The Environmental Protection Agency also stated that it is working toward consistent reporting on cost savings and avoidances in future reporting submissions and is finalizing a cost analysis methodology to be applied to its DCOI strategy. The agency further stated that it would ensure consistent use of the process for all reporting queries. We encourage the agency’s efforts to ensure consistent reporting across the various mechanisms that collect DCOI data. The Environmental Protection Agency‘s comments are reprinted in appendix X.

In written comments, the General Services Administration agreed with our recommendation that it take action to complete the missing elements in its strategic plan, including addressing any identified challenges, and submit the completed strategic plan to OMB. The agency stated that it had updated its DCOI strategic plan to complete the missing elements and had posted the plan on its public web page.

Our analysis of the agency's updated DCOI strategic plan determined that the plan includes the missing elements we identified during our review. We also found that the agency's CIO statement was available publicly on the agency's website. As a result, we have removed this recommendation.

The agency also agreed with our recommendation to ensure that the reporting of achieved data center consolidation and optimization cost savings and avoidances is consistent across all reporting mechanisms, including quarterly data submissions and agency DCOI strategic plans. The agency further described actions it planned to address the recommendation. Specifically, to better understand variances in OMB's cost model guidance, the agency said it will conduct a variance analysis of the differences in reported savings. Further, the agency said it will determine the most accurate representation of realized savings and will adjust its strategic plan and integrated data collection accordingly, making corrections in prior year projections and actual savings, as appropriate. We encourage the agency's efforts to ensure consistent reporting across the various mechanisms that collect DCOI data. The General Services Administration's comments are reprinted in appendix XI.

In written comments, the National Aeronautics and Space Administration agreed with our recommendation and stated that it would provide OMB with an update to the agency’s DCOI strategic plan that would address missing elements and any identified challenges. The National Aeronautics and Space Administration’s comments are reprinted in appendix XII.

In an e-mail received on March 23, 2017, a National Science Foundation GAO liaison said the agency had no comments on the draft report.

In written comments, the Nuclear Regulatory Commission stated that it generally agreed with our findings. However, the agency stated that it disagreed with our recommendation to take action to complete the missing elements in its DCOI strategic plan, including addressing any identified challenges, and to submit its completed strategic plan to OMB. Specifically, the agency asserted that its plan was completed and submitted via the process referenced by our recommendation and that OMB had assessed the agency's plan as including all information required by the schema provided by OMB. The Nuclear Regulatory Commission further stated that its CIO statement had been posted to an agency web page, along with a document that provided supplemental information beyond its strategic plan.

We acknowledge that the agency completed all the requirements found in the OMB schema; however, its plan did not include all the information required by OMB's August 2016 memorandum on the content of agencies' DCOI strategic plans. Notably, the Nuclear Regulatory Commission did not identify planned cost savings for fiscal years 2016-2018, actual cost savings for fiscal year 2016, or costs for consolidations and optimizations, as required by the memorandum. We also acknowledge that the agency posted its CIO statement to an agency web page, as required. However, OMB requires that this statement describe the reasons that an agency's DCOI strategic plan does not fully comply with all reporting requirements. Because we found that the Nuclear Regulatory Commission's plan did not have all required information, and because the agency's CIO statement did not address this missing information, we assessed the CIO statement as being only partially complete. All agencies in our review were similarly evaluated. Based on this analysis, we maintain that our findings are accurate and our recommendation appropriate.

The Nuclear Regulatory Commission also asserted that it had provided comments on a draft of the briefing, whereas we state that the agency did not. Specifically, the agency said that it provided written comments to us on November 30, 2016. However, those comments were in response to our November 21, 2016 request for factual corrections to the briefing and the agency's response was incorporated into the briefing, as appropriate. On December 6, 2016, we also requested that the agency comment on the briefing's conclusions and recommendations. We did not receive a response to that request, despite several efforts to obtain one. It was the status of this unanswered request that we summarized in the briefing passage cited by the Nuclear Regulatory Commission.

Additionally, the agency addressed planned actions to implement metering and monitoring capabilities for the data centers it intends to keep, in order to start collecting metrics that could be used to show future cost savings. We encourage these efforts, as the development of such metering and monitoring capabilities are key components of achieving the optimization and efficiency goals of DCOI. The Nuclear Regulatory Commission's written comments are reprinted in appendix XIII.

In written comments, the Office of Personnel Management agreed with our recommendation to take action to complete the missing elements in its DCOI strategic plan, including addressing any identified challenges, and submit the completed plan to OMB. In addition, the agency stated that it was in the process of updating its plan and would submit the plan to OMB once finalized. The agency later provided a copy of its strategic plan.

Our analysis of the agency's DCOI strategic plan determined that the plan contained all required elements. We also found that the agency's CIO statement was available publicly on the agency's website. As a result, we have removed this recommendation from the report. The Office of Personnel Management’s comments are reprinted in appendix XIV.

In an e-mail received on March 24, 2017, a program analyst with the Small Business Administration’s Office of Congressional and Legislative Affairs stated that the agency had no comments on the draft report and did not state whether the agency agreed or disagreed with the recommendations.

In written comments, the Social Security Administration agreed with our recommendation. The agency noted that it will continue to economize and evolve its data center optimization management and will continue to encourage open dialog and information exchange between agencies to achieve efficiencies and enhanced data center operations government-wide. The Social Security Administration’s comments are reprinted in appendix XV.

In written comments, the U.S. Agency for International Development did not agree or disagree with the recommendations, but described planned actions to address them. Specifically, the agency stated that it will, in accordance with OMB direction, take action to complete the missing elements in its DCOI strategic plan, including addressing any identified challenges, and submit the completed strategic plan to OMB. Further, the agency said that it will, in accordance with OMB, take action to ensure that the amounts of achieved data center cost savings and avoidances are consistent across all reporting mechanisms, including the quarterly data submissions and DCOI strategic plans. The U.S. Agency for International Development’s comments are reprinted in appendix XVI.

We are sending copies of this report to interested congressional committees, the Director of OMB, the secretaries and heads of the departments and agencies addressed in this report, and other interested parties. In addition, the report will be available at no charge on GAO’s website at http://www.gao.gov.

If you or your staffs have any questions about this report, please contact me at (202) 512-9286 or pownerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix XVII.

Appendix I: Briefing for Congressional Committees

Data Center Optimization: Agencies Need to Complete Plans to Address Inconsistencies in Reported Savings

Agency Comments and Our Evaluation The federal government’s demand for information technology (IT) is ever increasing. In recent years, as federal agencies have modernized their operations, put more of their services online, and improved their information security profiles, their need for computing power and data storage resources has grown. Accordingly, this growing demand has led to a dramatic rise in the number of federal data centers and a corresponding increase in operational costs. In response, the Office of Management and Budget’s (OMB) Federal Chief Information Officer (CIO) launched the Federal Data Center Consolidation Initiative (FDCCI) in 2010 to reduce the growing number of centers.

More recently, in August 2016, the Federal CIO launched the Data Center Optimization Initiative (DCOI) as a successor effort to FDCCI. According to OMB, this new initiative is intended to improve the performance of federal data centers in areas such as facility utilization and power usage. Among other things, DCOI requires 24 federal departments and agencies (agencies) to develop plans and report on strategies (referred to as DCOI strategic plans) to consolidate inefficient infrastructure, optimize existing facilities, improve security posture, and achieve cost savings.

With Congress recognizing the importance of reforming the government-wide management of IT, Federal Information Technology Acquisition Reform provisions (commonly referred to as FITARA) were enacted in December 2014 as a part of the Carl Levin and Howard P. ‘Buck’ McKeon National Defense Authorization Act for Fiscal Year 2015. Among other things, the law includes the following requirements related to federal data center consolidation:

Agencies shall annually report to OMB about federal data center inventories and strategies to achieve consolidation, including yearly calculations of investment and cost savings.

OMB shall develop and make publicly available a goal, broken down by year, for the amount of planned cost savings and optimization improvements achieved through FDCCI, and, for each year thereafter through October 1, 2018, compare progress against those goals.

In addition, the law includes a provision for GAO to annually review and verify the quality and completeness of agency federal data center inventories and strategies for consolidation.

Over the past several years, we have reported and testified that, while data center consolidation and optimization could potentially save the federal government billions of dollars, weaknesses exist in the execution and oversight of these efforts. For example, in March 2016, we reported that 10 agencies had not developed planned data center consolidation cost savings and avoidance targets for fiscal years 2016 through 2018, even though they had data center closures planned during this time period. As a result, we recommended that these agencies complete their planned cost savings and avoidance targets. Most agencies agreed with our recommendations or had no comments.

Our objectives for this study were to (1) review agencies’ data center closures to date and plans for further closures, (2) evaluate agencies’ progress in achieving data center consolidation savings and describe plans for future savings, and (3) evaluate the extent to which agencies’ annual strategic plans for data center consolidation and optimization are complete.

To accomplish the first objective, we obtained and analyzed data center inventory documentation from the 24 DCOI agencies in our review. To determine data center closures to date, we totaled agency reported closures from fiscal year 2010 through August 2016 and, to identify future closures, we totaled agency reported planned closures through fiscal year 2019. We also compared agencies’ completed and planned closures to OMB’s fiscal year 2018 consolidation goals, as documented in its August 2016 memorandum (M-16-19).

To verify the quality, completeness, and reliability of the agencies’ data center inventories, we compared information on data center closures completed and planned to similar information reported on OMB’s IT Dashboard—a public website that provides information on federal agencies’ major IT investments. We also checked for missing data and other errors, such as missing closure status information. Further, we obtained written responses from agency officials regarding actions taken to ensure the reliability of their inventory data, and discussed any discrepancies or potential errors identified to determine the causes or request additional information. We determined that the data were sufficiently complete and reliable to report on agencies’ consolidation progress and planned closures.

For our second objective, we obtained and analyzed cost savings and avoidance documentation from the 24 DCOI agencies required by OMB’s March 2013 and August 2016 memorandums. This documentation included the agencies’ quarterly reports of cost savings and avoidances posted to their digital strategy websites and their DCOI strategic plans. To determine cost savings achieved, we totaled agencies’ reported savings and avoidances from the start of fiscal year 2012 through August 2016, as found in the August 2016 quarterly reports posted on their digital strategy websites. To identify future planned savings, we totaled the agencies’ projected savings and avoidances from fiscal years 2016 through 2018, as reported in their DCOI strategic plans.

To assess the reliability of the agencies’ cost savings and avoidance data, we reviewed agency documentation for missing data or other errors (e.g., incorrect calculations). We also compared the cost savings and avoidances reflected in the agencies’ quarterly reports sent to OMB with the cost savings reported on OMB’s IT Dashboard. In addition, we obtained written responses from agency officials to confirm our understanding of their reported savings and avoidances. Finally, we compared agencies’ reported cost savings and avoidances with data from our March 2016 report on agencies’ data center consolidation efforts. As a result of our analyses, we found that certain agencies were reporting less cost savings and avoidances to OMB in their August 2016 quarterly reports and in their DCOI strategic plans compared to the amounts reported to us in November 2015. In certain cases, agencies were able to provide additional explanations for the differences in their savings. This issue is discussed in greater detail later in the briefing slides.

To address the third objective, we obtained and analyzed 20 of the 24 agencies’ DCOI strategic plans, which were required by FITARA and OMB’s August 2016 memorandum (M-16-19). This documentation included agencies’ strategic plan information publicly posted on agency-owned digital strategy websites, and additional agency- provided documentation of their data center consolidation and optimization strategic plans. We also obtained and reviewed agency CIO statements of compliance with FITARA and OMB’s guidance. To assess the completeness of agencies’ DCOI strategic plans, we compared information contained in the plans to the key elements required to be in the strategic plans, as called for in OMB’s memorandum implementing FITARA.

To assess the reliability of the DCOI strategic plans, we reviewed the agencies’ documentation to identify any missing data or errors. In addition, we obtained written responses from agency officials regarding the steps taken to ensure the accuracy and completeness of their strategic plan information. We also discussed with the officials any discrepancies or potential errors identified during our reviews of their strategic plans to determine the causes or request additional information. As a result of these efforts, we determined that the agencies’ strategic plan information was not reliable due to missing and incomplete data. This issue is discussed in greater detail later in the briefing slides.

We conducted this performance audit from July 2016 to December 2016 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

The 24 agencies participating in DCOI have collectively made progress on their data center closure efforts. Specifically, as of August 2016, the agencies had identified a total of 9,995 data centers, of which they reported closing 4,388. The Departments of Agriculture, Defense, Interior, and Treasury accounted for 84 percent of these closures. Further, agencies collectively reported that they are planning to meet OMB’s fiscal year 2018 consolidation targets to close 25 percent of tiered data centers and 60 percent of non-tiered centers. Specifically, as of August 2016, our analysis of agencies’ inventories showed that they were planning to close approximately 25 percent of the government-wide total of tiered data centers and approximately 66 percent of the government-wide total of non- tiered centers by the end of fiscal year 2018. When accounting for additional closures expected in fiscal year 2019, agencies have reported that they plan to close a total of 5,597 centers. Figure 1 provides a summary of agencies’ total data centers and completed and planned closures.

In total, 18 of the 24 agencies reported achieving an estimated $2.3 billion in cost savings and avoidances from their data center consolidation and optimization efforts from the start of fiscal year 2012 through August 2016. The Departments of Commerce, Defense, Homeland Security, and Treasury accounted for approximately $2.0 billion (or 87 percent) of the total. However, seven agencies are reporting less achieved cost savings and avoidances to OMB compared to savings amounts they previously reported to us in November 2015. Further, agencies are planning a total of approximately $378 million in savings between fiscal years 2016 and 2018, according to their DCOI strategic plans. This is about $3.6 billion less than the approximately $4.0 billion in fiscal year 2016 through 2018 planned savings that agencies reported to us in November 2015. This is also about $2.3 billion less than OMB’s $2.7 billion cost savings goal for agencies to achieve by the end of fiscal year 2018, as required by OMB M-16-19. This reduction in planned savings reflects 7 agencies reporting less planned cost savings and avoidances in their DCOI strategic plans compared to the savings amounts previously reported to us in November 2015, as well as the absence of cost savings information for 4 agencies that had not submitted their strategic plan. See figure 2 for a comparison of the amounts of cost savings and avoidances reported by agencies to OMB through their DCOI strategic plan and the amounts reported to us.

Of the 20 agencies that submitted their DCOI strategic plan as of December 2016, only 1—the National Science Foundation—had addressed all five elements required by the OMB memorandum implementing FITARA, while the remaining 19 either partially met or did not meet the requirements. For example, most agencies partially met or did not meet the requirements to provide information related to data center closures and cost savings metrics. See figure 3 for an assessment of the completeness of agencies’ data center optimization plans, by key element.

We also identified inconsistencies in the reporting of historical cost savings by 11 agencies across OMB’s two reporting mechanisms—the DCOI strategic plans and the quarterly data submissions. Specifically, of the 24 agencies, 9 consistently reported savings information in their DCOI strategic plan and their August 2016 quarterly reports to OMB, 11 agencies did not report consistent information, and 4 agencies did not submit their DCOI strategic plan. These inconsistencies resulted in a discrepancy totaling approximately $1.5 billion between the two sources. These inconsistencies were due, in part, to incomplete data in agencies’ DCOI strategic plans.

Agencies provided a variety of reasons for why their plans were incomplete, including the difficulty of reporting on fiscal year 2016 actual savings and closures when the data are not yet finalized and the receipt of incomplete data from some agency components. Completing these plans is increasingly important in light of the enactment of FITARA, which requires agencies to establish a multi-year strategy to consolidate and optimize their data centers, including performance metrics and cost savings goals. Moreover, as OMB’s data center consolidation and other related initiatives have been underway since 2010, agencies should have all the information necessary to complete their plans. Until agencies address the missing elements in their DCOI strategic plan, they will be at risk of not realizing anticipated savings, improved infrastructure utilization, and energy efficiency; they also may not be able to satisfy federal requirements regarding data center consolidation and optimization. Further, until OMB improves its oversight of agencies’ reporting of cost savings and agencies ensure that their savings information is consistent across all reporting mechanisms, the likelihood of further reporting errors is increased. Finally, until OMB improves its oversight of the information in agencies’ DCOI strategic plans, it may be challenged in demonstrating that DCOI is meeting its established objectives.

To better ensure that federal data center optimization efforts improve governmental efficiency and achieve cost savings, we are recommending that the Director of OMB direct the Federal CIO to provide the necessary oversight to (1) ensure that agencies complete their DCOI strategic plan in accordance with OMB’s guidance implementing FITARA and (2) ensure that agency reporting of achieved data center consolidation and optimization cost savings and avoidances is consistent across all reporting mechanisms, including quarterly data submissions and agency DCOI strategic plans.

We are also recommending that 23 agencies take action to complete the missing elements in their respective DCOI strategic plan, including addressing any identified challenges, and submit their completed strategic plan to OMB. Further, we are recommending that 11 agencies take action to ensure that the amounts of achieved data center cost savings and avoidances are consistent across all reporting mechanisms, including the quarterly data submissions and DCOI strategic plans.

We received comments on a draft of our briefing from OMB and 17 of the 24 agencies to which we made recommendations. In its comments, OMB neither agreed nor disagreed with our recommendations. Among the agencies, 4 agencies agreed with our recommendations, 1 (Agriculture) indicated that it did not agree with our recommendation, 3 commented on our findings but did not provide a position on the recommendations, and 8 stated that they had no comments. In addition, 1 agency provided only technical comments, while 2 agencies provided technical comments along with their other comments. All technical comments were incorporated as appropriate. We did not receive a response from 7 agencies.

The federal government’s increasing demand for IT has led to an increase in the number of federal data centers and a corresponding increase in operational costs. According to OMB, the federal government had 432 data centers in 1998, more than 1,100 in 2009, and 10,584 in November 2015. Operating such a large number of centers has been and continues to be a significant cost to the federal government, including costs for hardware, software, real estate, and cooling. For example, in 2007, the Environmental Protection Agency estimated that the electricity costs to operate federal servers and data centers across the government were about $450 million annually. According to the Department of Energy, a typical data center has 100 to 200 times the energy use intensity of a commercial building. In 2009, OMB reported that server utilization rates as low as 5 percent across the federal government’s estimated 150,000 servers were a factor driving the need to establish a coordinated, government-wide effort to improve the efficiency, performance, and environmental footprint of federal data center activities.

Background OMB and the Federal CIO Established FDCCI Concerned about the size of the federal data center inventory and the potential to improve the efficiency, performance, and the environmental footprint of federal data center activities, OMB, under the direction of the Federal CIO, established FDCCI in February 2010. This initiative’s four high-level goals were to reduce the overall energy and real estate footprint of government data centers; reduce the cost of data center hardware, software, and operations; increase the overall IT security posture of the government; and shift IT investments to more efficient computing platforms and technologies.

Under the first definition, OMB identified 2,094 data centers in July 2010. Under the subsequent definition, in December 2011, OMB estimated that there were a total of 3,133 federal data centers.

OMB, Implementation Guidance for the Federal Data Center Consolidation Initiative (Washington, D.C.: Mar. 19, 2012).

Background OMB’s Inventory of Federal Data Centers Has Grown Since 2011, the number of federal data centers reported by agencies has continued to grow. In March 2016, we reported that agencies had collectively identified a total of 10,584 data centers as of November 2015—an increase of about 7,500 data centers compared to OMB’s October 2011 estimate. According to the Federal CIO, the increase in data centers was primarily due to the expanded definition of a data center and improved inventory reporting by the agencies. Figure 4 shows the growth in the total number of reported federal data centers from 1998 to November 2015.

Appendix II: Comments from the Department of Commerce

Appendix III: Comments from the Department of Education

Appendix IV: Comments from the Department of Energy

Appendix V: Comments from the Department of Homeland Security

Appendix VI: Comments from the Department of Housing and Urban Development

Appendix VII: Comments from the Department of the Interior

Appendix VIII: Comments from the Department of State

Appendix IX: Comments from the Department of Veterans Affairs

Appendix X: Comments from the Environmental Protection Agency

Appendix XI: Comments from the General Services Administration

Appendix XII: Comments from the National Aeronautics and Space Administration

Appendix XIII: Comments from the Nuclear Regulatory Commission

Appendix XIV: Comments from the Office of Personnel Management

Appendix XV: Comments from the Social Security Administration

Appendix XVI: Comments from the U.S. Agency for International Development

Appendix XVII: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, individuals making contributions to this report included Dave Hinchman (Assistant Director), Justin Booth (Analyst-in-Charge), Gerard Aflague, Chris Businsky, Nancy Glover, Linda Kochersberger, Carlo Mozo, Jon Ticehurst, and Jonathan Wall.