Data Center Optimization: Additional Agency Actions Needed to Meet OMB Goals
Released: 2019-04-11T00:00:00


Highlights: section_title: Why GAO Did This Study
paragraphs: In December 2014, Congress enacted federal IT acquisition reform legislation that included provisions related to ongoing federal data center consolidation efforts. OMB's Federal Chief Information Officer launched DCOI to build on prior data center consolidation efforts; improve federal data centers' performance; and establish goals for inventory closures, cost savings and avoidances, and optimizing performance.

The 2014 legislation included a provision for GAO to annually review agencies' data center inventories and strategies. Accordingly, GAO's objectives were to (1) evaluate agencies' progress and plans for data center closures and cost savings; (2) assess agencies' progress against OMB's data center optimization targets; (3) and identify effective agency practices for achieving data center closures, cost savings, and optimization progress. To do so, GAO assessed the 24 DCOI agencies' data center inventories as of August 2018; reviewed their reported cost savings documentation; evaluated their data center optimization strategic plans; and assessed their progress against OMB's established optimization targets. GAO also solicited practices that selected agencies reported to be effective in meeting DCOI goals.

section_title: What GAO Found
paragraphs: The 24 agencies participating in the Office of Management and Budget's (OMB) Data Center Optimization Initiative (DCOI) reported mixed progress toward achieving OMB's goals for closing data centers and realizing the associated savings by September 2018. As of August 2018, 13 agencies reported that they had met, or had plans to meet, all of their OMB-assigned closure goals by the deadline. However, 11 agencies reported that they did not have plans to meet their goals. Further, 16 agencies reported that, as of August 2018, they had met, or planned to meet, their cost savings targets, for a total of $2.36 billion in cost savings for fiscal years 2016 through 2018. This is about $0.38 billion less than OMB's DCOI savings goal of $2.7 billion. This shortfall is the result of 5 agencies reporting less in planned cost savings and avoidances in their DCOI strategic plans, as compared to their savings targets established for them by OMB. Three agencies did not have a cost savings target and did not report any achieved savings.

In addition, the 24 agencies reported limited progress against OMB's five data center optimization targets for server utilization and automated monitoring, energy metering, power usage effectiveness, facility utilization, and virtualization. As of August 2018, the agencies reported that 3 had met three targets, 9 had met one target, and 10 met none of the targets. Two agencies did not have a basis to report on progress as they do not own any data centers. Further, as of August 2018, 20 agencies did not plan to meet all of OMB's fiscal year 2018 optimization goals. Specifically, only 2 agencies reported plans to meet all applicable targets; 6 reported that they did not plan to meet any of the targets (see figure).

We selected 6 agencies that had demonstrated success towards meeting their DCOI goals and those agencies reported a number of key practices that contributed to their efforts. The officials noted the importance of, among other things, obtaining executive leadership support for consolidation and optimization activities, employing an organization-wide communications plan, and focusing on data center closures. The officials also cited the use of past experience and lessons learned to inform improvements to future consolidation plans and processes.

section_title: What GAO Recommends
paragraphs: GAO is making 36 recommendations to 22 agencies to improve performance against established DCOI goals. Eleven agencies agreed with the recommendations, three did not fully agree, one disagreed, and seven neither agreed nor disagreed, as discussed in the report.

section_title: Letter
paragraphs: The federal government’s demand for information technology (IT) is ever  increasing. In recent years, as federal agencies have modernized their  operations, put more of their services online, and improved their  information security profiles, their need for computing power and data  storage resources has grown. Accordingly, this growing demand has led  to a dramatic rise in the number of federal data centers and a  corresponding increase in operational costs.

To reduce data center duplication and costs, the Office of Management  and Budget’s (OMB) Federal Chief Information Officer (CIO) launched  two initiatives. The first, launched in 2010, was the Federal Data Center  Consolidation Initiative (FDCCI), which aimed to reduce the number of  data centers that were outdated or duplicative. The second initiative—the  Data Center Optimization Initiative (DCOI)—was launched in August 2016  and superseded FDCCI. It shifted the focus to optimizing agencies’  remaining data centers by requiring, among other things, that agencies  consolidate inefficient infrastructure, optimize existing facilities, and  transition to more efficient infrastructure, such as cloud services.

Further, Congress recognized the importance of reforming the  government-wide management of IT and, in December 2014, enacted  Federal Information Technology Acquisition Reform provisions  (commonly referred to as FITARA) as a part of the Carl Levin and Howard  P. ‘Buck’ McKeon National Defense Authorization Act for Fiscal Year  2015. Among the requirements related to federal data center  consolidation, the law required:

Covered departments and agencies (agencies) to annually report to  OMB about federal data center inventories and strategies to achieve  consolidation, including yearly calculations of investment and cost  savings.

OMB to develop goals for the amount of planned cost savings and  optimization improvements that agencies are to achieve through  FDCCI. OMB is to make the goals publicly available and compare  progress against the goals.

In addition to these requirements, FITARA included a provision for GAO  to annually review and verify the quality and completeness of federal data  center inventories and strategies for consolidation submitted by the  agencies covered by the law. Accordingly, our specific objectives were to  (1) determine agencies’ progress on data center closures and the related  savings that have been achieved, and describe agencies’ plans for future  closures and savings; (2) assess agencies’ progress against OMB’s data  center optimization targets; and (3) identify effective agency practices for  achieving data center closures, cost savings, and optimization.

To review closures to date and plans for future closures, we obtained and  analyzed August 2018 data center inventory documentation from the 24  DCOI agencies. We compared information on these agencies’ completed  and planned data center closures to OMB’s fiscal year 2018 consolidation  goals, as documented in its August 2016 memorandum (M-16-19). We  determined the number of data centers that had been closed by adding  the closures from fiscal year 2010 through August 2018, as reported by  the agencies in their inventory submissions to OMB. We identified future  closures by counting data centers that agencies reported as planned  closures, as of August 2018 through fiscal year 2023.

To verify the quality, completeness, and reliability of the agencies’ data  center inventories, we compared the information on completed and  planned data center closures to similar information reported on OMB’s IT  Dashboard—a public website that provides information on federal  agencies’ major IT investments. We determined that the data were  sufficiently reliable to report on agencies’ consolidation progress and  planned closures.

To evaluate agencies’ progress in, and plans for, achieving data center  cost savings, we reviewed August 2018 cost savings and avoidance  documentation that the 24 DCOI agencies submitted in response to  OMB’s March 2013 PortfolioStat and August 2016 data center initiative  memorandums. This documentation included the agencies’ quarterly  reports of cost savings and avoidances posted to their digital services  websites and discussed in their DCOI strategic plans.

We determined cost savings achieved by adding agencies’ reported  savings and avoidances from the start of fiscal year 2012 through August  2018, as found in the August 2018 quarterly reports posted to the  agencies’ digital services websites. We identified planned savings by  totaling the agencies’ projected savings and avoidances from fiscal years  2016 through 2018, as reported in their DCOI strategic plans.

To assess the quality, completeness, and reliability of each agency’s data  center consolidation cost savings information, we used the latest version  of each agency’s update of the August 2018 quarterly cost savings report  and DCOI strategic plan. We also reviewed the quarterly reports and  DCOI strategic plans for missing data and other errors, such as missing  cost savings information.

In addition, we compared agencies’ reported cost savings and  avoidances with data from our most recently issued report on data center  consolidation. Further, we obtained written responses from agency  officials regarding the steps they took to ensure the accuracy and  reliability of their cost savings data. As a result, we determined that the  data were sufficiently reliable to report on agencies’ data center  consolidation cost-savings information.

To assess agencies’ progress against OMB’s data center optimization  targets, we obtained the August 2018 data center optimization progress  information of 22 DCOI agencies, as reported on the IT Dashboard. To  assess the agencies’ planned optimization progress, we obtained the  planned optimization performance from the 22 agencies’ DCOI strategic  plans. We then compared the agencies’ current and planned optimization  progress information to OMB’s fiscal year 2018 optimization targets, as  documented in its August 2016 memorandum.

To assess the reliability of agencies’ optimization progress information on  OMB’s IT Dashboard, we reviewed the information for errors or missing  data, compared agencies’ optimization progress information across  multiple reporting quarters to identify any inconsistencies in their reported  progress, and discussed with agency officials the steps they took to  ensure the accuracy and reliability of the reported optimization progress.  We determined that the data were sufficiently reliable to report on  agencies’ optimization progress.

In addition, to assess the reliability of the planned optimization milestones  in the DCOI strategic plans, we reviewed agencies’ documentation to  identify any missing or erroneous data. We also compared the planned  data center optimization milestones contained in agencies’ documentation  against current optimization progress information obtained from the IT  Dashboard; we then discussed any discrepancies or potential errors that  we identified with agency officials to determine the causes or request  additional information. As a result of these efforts, we were able to  determine whether each agency’s strategic plan information was  sufficiently reliable for reporting on plans to meet or not meet OMB’s fiscal  year 2018 optimization targets.

To identify effective agency practices for achieving data center closures,  cost savings, and optimization progress, we selected two of the highest- performing agencies for each of these three data center optimization  areas (closures, cost savings, and optimization performance) that we  reported on in our May 2018 report. For closures, we selected the  Departments of Agriculture (Agriculture) and Justice (Justice) as two  agencies that had, as of August 2017, reached or exceeded their DCOI  tiered and non-tiered data center closure targets.

For cost savings, we selected the Department of Commerce (Commerce)  and the General Services Administration (GSA) as two agencies reporting  some of the highest DCOI cost savings as of August 2017. For  optimization performance, we selected the Social Security Administration  (SSA) and the Environmental Protection Agency (EPA), the only two  agencies reporting, as of August 2017, that they had met more than half  of OMB’s optimization targets.

We sent each of these 6 agencies a list of open-ended questions  designed to solicit information on practices that the agencies found to be  effective in closing and optimizing data centers and identifying the  resulting cost savings. In doing so, our intent was to compile anecdotal  information that could assist other agencies struggling with DCOI  implementation. We do not consider the examples they provided to be  findings; nor should they be taken to be representative of all the agencies  participating in DCOI. Appendix I provides greater details regarding our  objectives, scope, and methodology.

We conducted this performance audit from April 2018 to April 2019 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: According to OMB, federal agencies reported that they operated 432 data  centers in 1998, 2,094 in July 2010, and 9,995 in August 2016.  Operating such a large number of centers has been, and continues to be,  a significant cost to the agencies. For example, in 2007, EPA estimated  that the annual cost for electricity to operate federal servers and data  centers across the government was about $450 million.

Further, according to the Department of Energy (Energy), a typical  government data center has 100 to 200 times the energy use intensity of  a commercial building. However, in 2009, OMB reported server  utilization rates as low as 5 percent across the federal government’s  estimated 150,000 servers. All of these factors contributed to OMB  recognizing the need to establish a coordinated, government-wide effort  to improve the efficiency, performance, and environmental footprint of  federal data center activities.
subsections: section_title: OMB and the Federal CIO Established FDCCI
paragraphs: Concerned about the size of the federal data center inventory and the  potential to improve the efficiency, performance, and environmental  footprint of federal data center activities, OMB’s Federal CIO established  FDCCI in February 2010. This initiative’s four high-level goals were to  reduce the overall energy and real estate footprint of government data  centers; reduce the cost of data center hardware, software, and  operations; increase the overall IT security posture of the government;  and shift IT investments to more efficient computing platforms and  technologies.

In February 2010, OMB required all of the agencies participating in the  FDCCI to submit a data center inventory and a consolidation plan. In  October 2010, OMB also clarified the definition of a data center and noted  that, for the purposes of FDCCI, a data center was to be defined as any  room used for the purpose of processing or storing data that is larger than  500 square feet and meets stringent availability requirements. Under this  definition, OMB reported that agencies had identified 2,094 data centers  as of July 2010.

However, in 2011, the Federal CIO expanded the definition to include a  facility of any size and OMB published its revised definition in March  2012. Based on the revised definition, OMB estimated that there were a  total of 3,133 federal data centers in December 2011. In addition, its goal  was to consolidate approximately 40 percent, or 1,253 of these data  centers, for a savings of approximately $3 billion by the end of 2015.

Figure 1 shows data center server racks at SSA’s National Support  Center in 2017.

The number of federal data centers reported by agencies has continued  to grow since 2011. In May 2018, we reported that agencies had  collectively identified a total of 12,062 data centers in their inventories as  of August 2017—an increase of about 9,000 data centers compared to  OMB’s October 2011 estimate. According to the Federal CIO, the  increase in the number of data centers was primarily due to the expanded  definition of a data center (discussed later in this report) and improved  inventory reporting by the agencies. See figure 2 for a depiction of the  increase in the number of data centers from 1998 through August 2018.

Further, OMB placed greater emphasis on data center optimization to  improve the efficiency of federal data centers when it issued  memorandum M-13-09 in March 2013. Specifically, OMB stated that, to  more effectively measure the efficiency of an agency’s data center  assets, agencies would also be measured by the extent to which their  primary data centers are optimized for total cost of ownership by  incorporating metrics for data center energy, facility, labor, and storage,  among other things.

Subsequently, in May 2014, OMB issued memorandum M-14-08, which  established a set of data center optimization metrics to measure agency  progress, along with target values for each metric. All agencies were  expected to achieve the target values by the end of fiscal year 2015.

section_title: IT Acquisition Reform Law Enhanced Data Center Consolidation and Optimization Efforts
paragraphs: Recognizing the importance of reforming the government-wide  management of IT, Congress enacted FITARA in December 2014.  Among other things, the law required agencies to:

Submit to OMB a comprehensive inventory of the data centers owned,  operated, or maintained by or on behalf of the agency.

Submit, by the end of fiscal year 2016, a multi-year strategy to  achieve the consolidation and optimization of the agency’s data  centers. The strategy was to include performance metrics that were  consistent with the government-wide data center consolidation and  optimization metrics.

Report progress toward meeting government-wide data center  consolidation and optimization metrics on a quarterly basis to OMB’s  Administrator of the Office of Electronic Government.

In addition, according to FITARA, the Office of Electronic Government at  OMB was to:

Establish metrics applicable to the consolidation and optimization of  data centers (including server efficiency), ensure that agencies’  progress toward meeting government-wide data center consolidation  and optimization metrics is made publicly available, review agencies’  inventories and strategies to determine whether they are  comprehensive and complete, and monitor the implementation of  each agency’s strategy.

Develop and make publicly available not later than December 19,  2015, a goal broken down by year for the amount of planned cost  savings and optimization improvements to be achieved through  FDCCI and, for each year thereafter until October 1, 2020, compare  reported cost savings and optimization improvements against those  goals.

section_title: OMB Established DCOI to Address FITARA Data Center Provisions
paragraphs: OMB issued memorandum M-16-19 in August 2016 to establish DCOI  and included guidance on how to implement the data center consolidation  and optimization provisions of FITARA. Among other things, the  guidance required agencies to consolidate inefficient infrastructure,  optimize existing facilities, improve their security posture, and achieve  cost savings. For example, each agency was required to maintain a  complete inventory of all data center facilities owned, operated, or  maintained by or on its behalf, and measure progress toward defined  optimization performance metrics on a quarterly basis as part of its data  center inventory submission.

OMB’s memorandum also directed each agency to develop a DCOI  strategic plan that defined its data center strategy for fiscal years 2016  through 2018. Among other things, this strategy was to include a timeline  for agency consolidation and optimization activities, with an emphasis on  cost savings and optimization performance benchmarks that the agency  could achieve between fiscal years 2016 and 2018. For example, each  agency was required to develop cost savings targets due to consolidation  and optimization actions and report any realized cost savings. OMB  required each agency to publicly post its DCOI strategic plan to its  agency-owned digital strategy website by September 30, 2016, and to  post subsequent strategic plan updates by April 14, 2017 and April 13,  2018.

Further, the memorandum stated that OMB was to maintain a public  dashboard (referred to as the IT Dashboard) to display government-wide  and agency-specific progress in areas such as planned and achieved  data center closures, consolidation-related cost savings, and data center  optimization performance information. In this regard, OMB began  including data center consolidation and optimization progress information  on the IT Dashboard in August 2016.

OMB’s memorandum also provided new guidance for the classification of  a physical data center and expanded the definition of a data center.  According to the revised definition, a room with at least one server that  provides services (whether in a production, test, staging, development, or  any other environment) should be considered a data center, while a room  containing only print servers, routing equipment, switches, security  devices (such as firewalls), or other telecommunication components, was  not to be considered a data center.

In light of this new definition, OMB directed each agency to perform a  comprehensive review of its data centers and maintain a complete and  updated data center inventory. Further, OMB directed each agency to  categorize each of its data centers as either a tiered data center or a non- tiered data center. OMB’s memorandum defined a tiered data center as  one that uses each of the following: a separate physical space for IT infrastructure; an uninterruptible power supply; a dedicated cooling system or zone; and a backup power generator for a prolonged power outage.

According to the memorandum, all other data centers were to be  considered non-tiered.

Moreover, OMB guidance included a series of performance metrics in the  areas of data center closures, cost savings, and optimization progress.

Data center closures: According to the guidance, agencies were to  close at least 25 percent of tiered data centers government-wide,  excluding those approved as inter-agency shared services providers,  by the end of fiscal year 2018. Further, agencies were to close at  least 60 percent of non-tiered data centers government-wide by the  end of fiscal year 2018. OMB’s guidance further notes that, in the long  term, all agencies should continually strive to close all non-tiered data  centers, noting that server rooms and closets pose security risks and  management challenges and are an inefficient use of resources.

Cost savings: According to the guidance, agencies were to reduce  government-wide annual costs attributable to physical data centers by  at least 25 percent, resulting in savings of at least $2.7 billion for fiscal  years 2016 through 2018.

Data center optimization: According to the guidance, agencies were to  measure progress against a series of new data center performance  metrics in the areas of server utilization, energy metering, power  usage, facility utilization, and virtualization. Further, OMB’s guidance  established target values for each metric that agencies were to  achieve by the end of fiscal year 2018.

OMB’s guidance further noted that agency progress against these  performance metrics was to be measured by OMB on a quarterly basis,  using agencies’ data center inventory submissions and OMB-defined  closures, cost savings, and optimization targets.

section_title: OMB Published Proposed Changes to DCOI in November 2018
paragraphs: In November 2018, OMB published proposed changes to DCOI for public  comment. The changes focus federal consolidation and optimization  efforts on agencies’ larger, tiered data centers and also de-emphasize the  consolidation of non-tiered facilities and other smaller spaces. The draft  guidance also revises the classification of data centers and data center  optimization metrics.

The draft guidance redefines a data center as a purpose-built, physically  separate, dedicated space that meets certain criteria. Similarly, OMB does not plan to continue to report on spaces not designed to be data  centers. According to the draft, OMB also plans to work with agencies to  set agency-specific goals for data center closures and cost savings and to  update these targets from those set in OMB’s August 2016 memorandum  to match agencies’ current status and progress.

Additionally, the proposed changes to DCOI make several changes to the  metrics currently used by agencies to monitor the performance of their  data centers. Specifically, of the five metrics currently in use (described in  detail later in this report), OMB proposes updating three, removing two,  and adding one new metric.

The draft guidance states that public comments will be collected through  the end of December 2018, but does not provide a date for when the  proposed changes will be finalized and implemented. However, the draft  does state that the new guidance will sunset on September 30, 2020, a  date that coincides with the extension of FITARA’s data center provisions.

section_title: Agencies Have Taken Limited Action to Address Prior GAO Recommendations
paragraphs: Since the enactment of FITARA in December 2014, we have reviewed  and verified annually for the quality and completeness of each agency’s  (covered by the law) inventory and DCOI strategy. We have also  published reports documenting the findings from each of our reviews. In  addition, we have examined and reported on agencies’ efforts to optimize  their data centers, as well as the challenges encountered and successes  achieved.

In a report that we issued in March 2016, we noted that agencies had  reported significant data center closures—totaling more than 3,100  through fiscal year 2015—with the Departments of Agriculture, Defense  (Defense), the Interior (Interior), and the Treasury (Treasury) accounting  for 84 percent of the total. Although the agencies fell short of OMB’s fiscal year 2015 consolidation goal, their plans identified about 2,100  additional centers planned for closure through fiscal year 2019.

Agencies also reported significant consolidation cost savings and  avoidances—totaling about $2.8 billion through fiscal year 2015 and  expected to increase to over $8.0 billion in future years. The Departments  of Commerce, Defense, Homeland Security (DHS), Transportation  (Transportation), and the Treasury accounted for 96 percent of the total  planned savings.

However, we pointed out that many agencies lacked complete cost  savings goals for the next several years despite having closures planned.  In addition, we reported that 22 agencies had made limited progress  against OMB’s fiscal year 2015 data center optimization performance  metrics, such as the utilization of data center facilities. Accordingly, we  recommended that the agencies take actions to complete their cost  savings targets and improve optimization progress. As of December  2018, 18 of the 32 recommendations from this report had yet to be fully  addressed.

In May 2017, we reported that the agencies were reporting significant  data center closures—totaling more than 4,300 through August 2016— with Agriculture, Defense, Interior, and the Treasury accounting for 84  percent of the total. The agencies’ plans for 2016 had identified more  than 1,200 additional centers for closure through fiscal year 2019.

Agencies also reported significant consolidation and optimization cost  savings and avoidances, which totaled about $2.3 billion through August  2016. However, reductions in the amount of achieved savings reported to  OMB, particularly by the Treasury, resulted in a net decrease of more  than $400 million in these savings, compared to amounts we previously  reported as planned in 2015.

Further, our report noted that, as of December 2016, agencies’ total  planned cost savings of about $656 million were more than $3.3 billion  less compared to the amounts that we reported in 2015, and more than  $2 billion less than OMB’s fiscal year 2018 cost savings goal of $2.7  billion. This reduction in planned savings was the result of eight agencies  reporting less in planned cost savings and avoidances in their DCOI  strategic plans compared to the savings amounts previously reported to  us in November 2015. The reduction also reflected the absence of cost  savings information for one agency (Defense) that did not submit its  strategic plan in time for our review.

In addition, our May 2017 report identified weaknesses in agencies’ DCOI  strategic plans. Of the 23 agencies that had submitted their strategic  plans at the time of our review, 7 agencies—Agriculture, the Department  of Education (Education), DHS, and the Department of Housing and  Urban Development (HUD); GSA; the National Science Foundation  (NSF); and the Office of Personnel Management (OPM)—had addressed  all five required elements of a strategic plan, as identified by OMB (such  as providing information related to data center closures and cost savings  metrics). The remaining 16 agencies that submitted their plans either  partially met or did not meet the requirements. We also pointed out that  there were inconsistencies in the reporting of cost savings in the strategic  plans of 11 agencies.

Given these findings, we recommended that OMB improve its oversight of  agencies’ DCOI strategic plans and their reporting of cost savings and  avoidances. We also recommended that 16 agencies and Defense (which  did not submit a plan in time for our review) complete the missing  elements in their strategic plans, and that 11 agencies ensure the  reporting of consistent cost savings and avoidance information to OMB.  As of December 2018, 10 of the 30 recommendations had not been fully  addressed.

In a subsequent report that we issued in August 2017, we noted that 22 of  the 24 agencies required to participate in the OMB DCOI had reported  (collectively) limited progress against OMB’s fiscal year 2018  performance targets for the five optimization metrics. The 2 remaining  agencies, Education and HUD, did not own any data centers and,  therefore, did not have a basis to report on progress. Specifically, for  each of the five targets, no more than 5 agencies reported that they had  met or exceeded that specific target. We reported that this limited  progress against OMB’s optimization targets was due, in part, to agencies  not fully addressing our prior recommendations in this area.

In addition, we noted in the report that most agencies had not yet  implemented automated monitoring tools to measure server utilization, as  required by the end of fiscal year 2018. Specifically, 4 agencies reported  that they had fully implemented such tools, 18 reported that they had not  yet done so, and 2 did not have a basis to report on progress because  they did not own any data centers. Accordingly, we recommended that  OMB require that agencies include plans, as part of existing OMB  reporting mechanisms, to implement automated monitoring tools at their  agency-owned data centers. We also recommended that the 18 agencies  that did not have fully documented plans take action, within existing OMB  reporting mechanisms, to complete plans describing how they intended to  achieve OMB’s requirement to implement automated monitoring tools at  all agency-owned data centers by the end of fiscal year 2018. As of  December 2018, none of our 19 recommendations had been fully  addressed.

Most recently, in May 2018, we noted that the 24 agencies participating in  DCOI had reported mixed progress toward achieving OMB’s goals for  closing their data centers by September 2018. Thirteen agencies reported  that they had either already met, or planned to meet, all of their OMB- assigned goals by the deadline. However, 4 agencies reported that they  did not have plans to meet all of their assigned goals and 2 agencies  were working with OMB to establish revised targets.

With regard to agencies’ progress in achieving cost savings, 20 agencies  reported, as of August 2017, that they had achieved $1.04 billion in cost  savings for fiscal years 2016 and 2017. In addition, the agencies’ DCOI  strategic plans identified an additional $0.58 billion in planned savings— for a total of $1.62 billion for fiscal years 2016 through 2018. This total  was approximately $1.12 billion less than OMB’s DCOI savings goal of  $2.7 billion. This shortfall was the result of 12 agencies reporting less in  planned cost savings and avoidances in their DCOI strategic plans, as  compared to the savings targets established for them by OMB.

In addition, the 24 agencies reported limited progress against OMB’s five  data center optimization targets for server utilization and automated  monitoring, energy metering, power usage effectiveness, facility  utilization, and virtualization. As of August 2017, 1 agency had met four  targets, 1 agency had met three targets, 6 agencies had met either one or  two targets, and 14 agencies reported meeting none of the targets.  Further, as of August 2017, most agencies were not planning to meet  OMB’s fiscal year 2018 optimization targets. Specifically, 4 agencies  reported plans to meet all of their applicable targets by the end of fiscal  year 2018; 14 reported plans to meet some of the targets; and 4 reported  that they did not plan to meet any targets.

Because GAO had made a number of recommendations to OMB and the  24 DCOI agencies to help improve the reporting of data center-related  cost savings and to achieve optimization targets, we did not make new  recommendations and noted that, as of March 2018, 74 of the 81 prior  recommendations had not been fully addressed. While agencies have  made considerable progress, as of December 2018, 47 of the 81  recommendations had not been fully addressed.

section_title: Agencies Reported Mixed Results in Efforts and Plans to Meet OMB’s Targets for Data Center Closures and Cost Savings
paragraphs: According to OMB guidance, agencies were expected to close at least 25  percent of tiered data centers government-wide, by the end of fiscal  year 2018. In addition, agencies were to close at least 60 percent of  non-tiered data centers government-wide by this same deadline. Further,  agencies were expected to reduce government-wide annual costs  attributable to physical data centers by a least 25 percent by the end of  fiscal year 2018, resulting in savings of at least $2.7 billion.
subsections: section_title: About Half of the Agencies Planned to Meet OMB’s Targets for Data Center Closures
paragraphs: The 24 agencies reported mixed results regarding their data center  closure progress and plans, when compared with OMB’s goal for each  agency to close at least 25 percent of their tiered data centers and at  least 60 percent of their non-tiered centers. Specifically, as of August  2018, 13 agencies reported that they had already met the goal of closing  25 percent of their tiered data centers, another 3 agencies reported that  they planned to meet the goal by the end of fiscal year 2018, and 6  agencies reported that they did not plan to meet the goal.

Further, as of August 2018, 11 agencies reported that they had already  met the goal for closing 60 percent of their non-tiered centers, 3 agencies  reported that they planned to meet the goal by the end of fiscal year  2018, and 9 agencies reported that they did not plan to meet the goal by  the end of fiscal year 2018. Table 1 displays a breakdown of the number  of reported tiered and non-tiered data centers and completed and  planned closures by agency, as of August 2018.

As shown in the figure below, the 24 agencies reported a total of 6,250  data center closures as of August 2018, which represented about half of  the total reported number of federal data centers. In addition, the  agencies planned 1,009 closures by the end of fiscal year 2018, with an  additional 191 closures planned through fiscal year 2023 for a total of  1,200 more closures. This would further reduce the number of open data  centers to about 39 percent of the number reported in the agencies’  inventories. Figure 3 provides a summary breakdown of agencies’ data  center inventories that were closed, planned for closure, or not planned  for closure, as of August 2018.

As noted, while about half of the agencies had met, or had planned to  meet, their OMB targets as of August 2018, the other half planned to miss  one or both of them. Officials from the 11 agencies that did not plan to  meet one or both of their closure goals provided various reasons for why  they had not planned to do so. For example, several agencies indicated  that they were seeking revised closure goals because they viewed their  goals as unattainable. Specifically, officials from Interior’s Office of the  CIO stated that a number of the department’s non-tiered data centers  were either mission-critical or not cost effective to close. Thus, the  officials said Interior was working with OMB to establish a revised closure  goal.

Similarly, Transportation’s Director for IT Compliance stated that the  department was working with OMB to establish a revised closure goal.  The department reported having 186 tiered data centers in Federal  Aviation Administration control towers that it believes should be excluded  from its count of data centers when OMB sets the department’s goal for  closures. In addition, officials in Defense’s Office of the CIO stated that  the OMB closure targets for the department were based on including  special purpose processing nodes that are mission critical and, therefore,  are not subject to being closed. The officials noted that the department  intends to continue operating its enterprise data centers, close its smaller  data centers, and work with OMB to remove the special purpose  processing nodes from DCOI consideration.

When OMB launched DCOI in 2016, agencies originally had until the end  of fiscal year 2018 to meet OMB’s stated time frame for closing their data  centers. However, the extension of FITARA’s data center consolidation  and optimization provisions through fiscal year 2020, and OMB’s planned  revisions to DCOI goals, provide the 11 agencies that had not planned to  meet one or both of OMB’s closure targets with additional time to meet  their goals. Until these agencies take action to close enough data centers  to meet OMB’s targets, they may not realize the efficiencies and cost  savings that were expected from DCOI.

section_title: Almost Two-thirds of Agencies Planned to Meet OMB-Assigned Savings Targets
paragraphs: Since 2013, federal agencies have been required to report on data center  cost savings, with guidance from OMB regarding how agencies were to  report cost savings and avoidances. Specifically, the guidance required  agencies to report both data center consolidation cost savings and  avoidances, among other areas, as part of a quarterly reporting process.  FITARA also called for each agency to submit a multi-year strategy for  achieving the consolidation and optimization of data centers that included  year-by-year quarterly calculations of investment and cost savings  through fiscal year 2018, which has now been extended to 2020.

In addition, in August 2016, OMB M-16-19 provided guidance on how  agencies should implement the requirements of FITARA. Specifically,  agencies were to develop a strategic plan that included information on  historical cost savings and avoidances due to data center consolidation  and optimization through fiscal year 2015. This guidance stated that  agency strategic plans were also to include year-by-year calculations of  target and actual agency-wide spending and cost savings on data centers  from fiscal years 2016 through 2018. Further, the guidance established a  DCOI combined cost savings goal of $2.7 billion for all federal agencies to  achieve from fiscal years 2016 through 2018. This overall goal was then  broken down into agency-specific targets on the IT Dashboard.

In August 2018, 22 agencies reported through the quarterly reporting  process that they had achieved $1.94 billion in cost savings for fiscal  years 2016 through 2018, while 2 agencies reported that they had not  achieved any savings. Further, 21 agencies identified an additional $0.42  billion planned through fiscal year 2018, for a total of $2.36 billion in  planned savings from fiscal years 2016 through 2018. Nevertheless, this  total is about $0.37 billion less than OMB’s goal of $2.7 billion for overall  DCOI savings. Figure 4 compares the total achieved savings as reported  by the 24 agencies for fiscal years 2016 through 2018 and the agencies’  additional planned savings through 2018 to OMB’s DCOI savings goal for  fiscal years 2016 through 2018.

The 24 participating DCOI agencies had achieved $1.94 billion in savings  as of August 2018. In addition, agencies identified an additional $0.43  billion, for a difference of $0.37 billion between planned and achieved  savings from fiscal years 2016 through 2018. Table 2 provides specific  data related to each agency’s total planned savings, total achieved  savings, and additional planned savings through 2018.

As shown in table 2, 13 agencies reported that they had met or planned  to meet or exceed their OMB targets, and 3 agencies that did not have an  OMB target also identified achieved savings. In contrast, 5 agencies  reported that they did not plan to meet their targets. Three agencies did  not have a savings target and did not report any achieved savings.

Agencies provided various reasons for why they did not plan to meet their  savings targets. For example, the Department of Veterans Affairs (VA)  reported that the implementation of its DCOI-related projects in fiscal  years 2016 through 2018 was dependent on funding approval and might  not result in cost savings and avoidances until later in the projects’ life  cycles (i.e., fiscal year 2019 or later). In another example, GSA stated  that the OMB target may be difficult for the agency to reach due, in part,  to the methods OMB used to set the target for fiscal years 2016 through  2018. According to GSA, OMB used the data that GSA had reported on  the IT Dashboard regarding the agency’s expenditure for data center  infrastructure and reduced that amount by 25 percent.

Agencies have now been working toward OMB’s DCOI savings goals  since fiscal year 2016; however, almost half of the agencies are still not  planning to meet OMB’s targets. Until agencies plan to meet and achieve  OMB’s data center-related savings targets, they will likely not realize the  expected financial benefits from DCOI.

section_title: Most Agencies Continued to Report Limited Progress Toward Meeting Optimization Metrics Targets
paragraphs: FITARA required OMB to establish metrics to measure the optimization of  data centers, including server efficiency, and to ensure that agencies’  progress toward meeting the metrics is made public. Pursuant to FITARA,  OMB established a set of five data center optimization metrics intended to  measure agencies’ progress in the areas of server utilization and  automated monitoring, energy metering, power usage effectiveness,  facility utilization, and virtualization. According to OMB, while the server  utilization and automated monitoring metric applied to agency-owned  tiered and non-tiered data centers, the four remaining metrics applied  only to agency-owned tiered centers.

OMB’s memorandum also established a target value for each of the five  metrics, which agencies were expected to achieve by the end of fiscal year 2018. OMB measures agencies’ progress against the optimization  targets using the agencies’ quarterly data center inventory submission  and publicly reports this information on its IT Dashboard. Table 3 provides  a description of the five data center optimization metrics and target  values.

As of August 2018, most (22 of the 24) DCOI agencies continued to  report limited progress in meeting OMB’s fiscal year 2018 data center  optimization targets identified on the IT Dashboard. The remaining 2  agencies—Education and HUD—reported that they did not have any  agency-owned data centers in their inventory and, therefore, did not have  a basis to measure and report optimization progress.

With regard to the data center optimization targets, agencies reported the  greatest progress against two metrics: power usage effectiveness and  virtualization metrics. Specifically, 8 agencies reported that they had met  OMB’s target for power usage effectiveness and 6 agencies reported that  they had met the target for virtualization. However, for the energy  metering, facility utilization, and server utilization and automated  monitoring metrics, no more than 3 agencies reported meeting each.  Figure 5 summarizes the 24 agencies’ progress in meeting each  optimization target, as of August 2018.

As of August 2018, NSF, SSA, and EPA reported the most progress  against OMB’s metrics among the 22 agencies with a basis to report— each met 3 targets. Nine agencies reported that they had met only one  target, and 10 agencies reported they had not met any of the targets.

Further, OMB began requiring the implementation of automated  monitoring tools in August 2016; however, as of August 2018, of the 22  agencies with a basis to report, 5 reported that they had either not  implemented the tools at any data centers, or had experienced  shortcomings in their implementation. For example, the Department of  State (State) reported that it had limited centralized monitoring capability  and is installing automated monitoring tools in several phases.

Thus, these 5 agencies were not able to report any progress against  either or both of the server utilization or power usage effectiveness  metrics because their data centers lacked the required monitoring tools to  measure progress in these areas. The remaining 17 agencies reported  that they had implemented the tools in at least one data center. Table 4  depicts the performance of the agencies in meeting OMB targets for data  center optimization, as of August 2018.

As of August 2018, multiple agencies had made changes to their data  center inventory and operational environment, such as closing all agency- owned tiered data centers or implementing automated monitoring tools.  These changes impacted which metrics were applicable or an agency’s  ability to report on the status of its optimization metrics. For example,  GSA reported that it no longer had any agency-owned tiered data  centers, and, therefore, did not have a basis to report on four of the five  optimization metrics. Additionally, NSF, which previously had only owned  one non-tiered data center, migrated from the non-tiered center to a tiered  data center as part of its headquarters relocation. Accordingly, NSF  began reporting on the metrics applicable to its tiered facility.

Further, the Nuclear Regulatory Commission (NRC) did not report on  power usage effectiveness due to delays in awarding a new contract that  was to include monitoring tools that would impact the ability to report on  this metric. Agency officials stated that NRC plans to have the monitoring  tools in place during fiscal year 2019. Overall, these changes since last  year’s report have resulted in no significant changes to the progression of  these agencies on their optimization metrics.

In addition, agencies’ limited progress against OMB’s optimization targets  was due, in part, to not fully addressing our prior recommendations in this  area. As previously mentioned, in March 2016, we reported on  weaknesses in agencies’ data center optimization efforts, including that  22 agencies did not meet OMB’s fiscal year 2015 optimization targets.  We noted that this was partially due to the agencies having challenges in  optimizing their data centers, including in their decentralized  organizational structures that made consolidation and optimization  difficult, and in competing priorities for resources. In addition,  consolidating certain data centers was problematic because the volume  or type of information involved required the data center to be close in  proximity to the users. Accordingly, we recommended that the agencies  take action to improve optimization progress, to include addressing any of  the identified challenges. Most agencies agreed with our  recommendations or had no comments. However, as of December 2018,  only 4 of the 22 agencies had fully addressed them.

The continuing shortcomings in data center optimization can also be  attributed, in part, to agencies viewing OMB’s optimization metric targets  as unrealistic. For example, Transportation stated in its DCOI strategic  plan that it could not meet multiple optimization metrics due to funds not  being available and competing priorities. In addition, Treasury indicated in  its DCOI strategic plan that it struggles to report on automated monitoring  because many of its data centers do not have the ability to centrally  aggregate and report on central processing unit data. Further, DHS  officials noted that it has 7 smaller tiered data centers where it has  determined that it is not cost effective to equip those centers with the  tools needed to report on metrics such as power usage effectiveness.  Given these types of challenges, the targets for each optimization metric  may not be realistic for every agency. Unless agencies take action to  meet the applicable OMB optimization metrics, their data centers may not  operate efficiently enough to provide expected cost savings.
subsections: section_title: Only Two Agencies Planned to Meet OMB’s Fiscal Year 2018 Optimization Targets
paragraphs: In addition to reporting current optimization progress on the IT  Dashboard, OMB required agencies to include in their DCOI strategic  plans planned performance levels for fiscal year 2018 for each  optimization metric.

However, according to the 24 agencies’ DCOI strategic plan information  as of August 2018, only 2—Commerce and the U. S. Agency for  International Development (USAID)—reported plans to fully meet their  applicable targets by the end of fiscal year 2018. Of the remaining  agencies, 14 reported plans to meet some, but not all, of the targets; 6  reported that they did not plan to meet any targets, and—as already  discussed—Education and HUD did not have a basis to report planned  optimization milestones because they did not report having any agency- owned data centers. Figure 6 summarizes agencies’ progress, as of  August 2018, in meeting OMB’s optimization targets and planned  progress to be achieved by September 2018.

At the time of our review, only two agencies planned to meet all of their  applicable targets, and it was doubtful that the agencies would be able to  achieve OMB’s collective optimization target of at least $2.7 billion in cost  savings by the end of fiscal year 2018. Until the remaining agencies  take the steps necessary to meet their optimization targets, it is unlikely  that these agencies will achieve the expected benefits of optimization and  the resulting cost savings.

section_title: Selected Agencies Highlighted Successful DCOI Practices
paragraphs: As we noted previously in this report, many agencies have reported  challenges that have hindered their efforts to meet OMB’s DCOI targets.  However, a number of agencies have also reported success in meeting  OMB’s targets ahead of DCOI’s end of fiscal year 2018 deadline. As  noted in our methodology section, six agencies that were among the best  performers in achieving data center closures, cost savings, and  optimization performance reported a number of key practices that had  contributed to their success. These practices were: obtaining executive leadership support for consolidation and using experiences and lessons learned to refine consolidation  increasing the use of cloud and shared services to consolidate or  optimize data center operations; emphasizing closing data centers to meet OMB targets and achieve  increasing the use of virtualization to optimize data centers; and employing an organization-wide communications plan to facilitate  adoption of consolidation and optimization activities.
subsections: section_title: Obtaining Executive Leadership Support for Consolidation and Optimization Activities
paragraphs: Five of the six agencies (Agriculture, Commerce, Justice, EPA, and GSA)  reported that their success in consolidation and optimization activities was  due to obtaining support from executive leadership for the agency’s  consolidation efforts. Each agency obtained sponsorship and support  from its executive leadership (e.g., Deputy Secretary or agency CIO),  such as through a memorandum or policy that directed all agency offices  to participate in, or comply with, the consolidation effort. For example,

The Deputy Secretary for Agriculture issued a memorandum in 2017  that, among other things, declared the department’s intent to  consolidate from 39 data centers down to 2 by the end of 2019.  According to officials in the Office of the CIO, this memorandum from  the Secretary’s office focused all data center owners on the same  project task of reducing the data center inventory.

The Commerce CIO and the department’s CIO Council provided  overall governance through organizational policies, processes, and  procedures for the department’s data center consolidation effort.  Leveraging this departmental guidance, each component of  Commerce developed its own consolidation plan that identified  specific approaches and activities. Using these plans, the department  and its components focused on reducing spending on redundant  software, infrastructure, and data center operations.

The Deputy Attorney General issued a memorandum in 2014 to the  heads and CIOs of all components. This memorandum formally  established Justice’s Data Center Transformation Initiative,  established the Department Program Review Board to provide  oversight for the initiative, and also directed the consolidation of all  data centers into 3 enterprise facilities. In addition, Justice’s CIO  issued a memorandum to component CIOs that provided additional  details on how to execute planned activities and established further  governance associated with the initiative. These memoranda provided  clear leadership buy-in and support for the department’s data center  consolidation and optimization activities that could be used to resolve  any challenges or issues at the departmental level.

EPA attributed much of its DCOI success to a top-down approach  from its CIO office, saying that such support was critical to achieve  data center closures. For example, EPA leadership decided to adopt  and enforce geographical consolidation of data centers within major  areas to minimize costs of consolidation while still meeting closure  objectives. In doing so, the agency leadership provided clear direction  and support for the agency’s consolidation effort by adopting the  strategy to consolidate data centers within specific geographic  regions.

GSA reported that it obtained leadership commitment that made its  data center consolidation and optimization activities a priority. The  agency noted that having strong CIO and executive leadership was  important for sponsoring technology modernization. As a result of the  buy-in, the agency reported that it had minimized resistance to  change and improved acceptance of its consolidation and optimization  activities.

section_title: Using Experiences and Lessons Learned to Refine Consolidation Planning
paragraphs: Four agencies (Agriculture, Commerce, Justice, and SSA) reported that  their success with consolidation and optimization activities was due to the  use of a refined consolidation plan or process. Each of these agencies  developed an initial consolidation plan or process for closing data centers,  and then refined their procedures based on their experiences and lessons  learned as data centers were closed. For example,

Agriculture developed a set of streamlined processes to facilitate  DCOI closures that were based on the experiences gained from  successful data center closures under FDCCI. The set of processes  consisted of 5 steps:

The planning step included the discovery and documentation of all  data center assets, including applications and IT hardware, in a  given data center. In addition, this step involved identifying the  necessary resources to move the applications and associated  data to a target data center.

The preparation step included identification of the target data  center and development of a project schedule.

The data migration step included moving both applications and  data to the target data center or cloud-services, as planned.

The testing step included ensuring the applications and data that  were moved were integrated into the target data center, and  functional testing to ensure that the applications worked and data  was accessible.

The application cutover step included putting the migrated  applications and data into operation and closing the original data  center.

Using and refining this set of processes allowed the department to  become more efficient in closing its data centers. After closing 46 data  centers in fiscal years 2011 through 2014, the department closed  2,185 data centers over the next 2 years. In total, Agriculture reported  that it had closed 2,253 data centers as of August 2018.

Commerce established departmental guidance and then each  departmental component leveraged that guidance to develop its own  consolidation plan. The plans identified specific approaches and  activities intended to achieve the stated goals and milestones.  According to Commerce, the department and its components  leveraged their IT planning processes and established IT governance  to, among other things, reduce spending on redundant commodity  software, infrastructure, and operations.

Justice’s Office of the CIO developed a master plan for the  department’s data center consolidation effort in June 2015. The plan  included a planning framework, transformation approach, and a  master schedule for data center moves and closures. It also included  process steps similar to those used by Agriculture. Further, Justice’s  plan noted that the department would use its initial closure efforts to  gain experience and to refine its plans. Justice reported that it used  the plan’s schedule and semi-monthly progress reports to ensure that  consolidation activities stayed on schedule, or the department could  make adjustments as needed. As a result, the department closed 84  of its 110 data centers and achieved more than $128 million in cost  savings and avoidances as of August 2018.

SSA used a project management framework process and controls that  it believed efficiently addressed requirements, critical path, and risk  management. In addition, SSA reported that it used an incremental  development approach to its data center optimization plans, with each  project expected to accomplish specific tasks that would lead to  another project. Accordingly, SSA noted that the agency used a  multi-year plan with many initiatives focused on specific goals. Using  this approach, the agency successfully moved SSA’s operations and  infrastructure from an older facility to the newly-built National Support  Center. The agency reports that this facility is state-of-the-art and  provides similar capabilities and efficiencies to major cloud service  providers.

section_title: Increasing the Use of Cloud and Shared Services to Consolidate or Optimize Data Center Operations
paragraphs: Three agencies (Commerce, GSA, and SSA) also attributed their success  in consolidation and optimization activities to increasing their agency’s  use of cloud and shared services. In doing so, each agency emphasized  the move of data center assets and systems to cloud services to optimize  their data centers and reduce costs. For example,

Commerce identified moving to cloud services and utilizing shared  services as being most effective in closing data centers. As an  example, the department cited the National Oceanic and Atmospheric  Administration’s (NOAA) “cloud-first” policy that emphasized using  cloud services rather than an agency-owned physical data center  whenever feasible. The agency attributed its ability to handle  increased traffic as an operational benefit of its increased use of cloud  services. For example, NOAA did not have the capacity in its agency- owned facilities to meet the computing demands and requirements of  a sudden increase in web traffic on the websites for NOAA and the  National Hurricane Center, such as during Hurricanes Irma and  Harvey in 2017. Commerce stated that using cloud services allowed  NOAA to handle 4.7 billion page hits during Hurricane Harvey over a  6-day span, ensuring the websites were not adversely impacted by  the increase in traffic.

GSA reported that it focused on moving services from agency-owned  tiered and non-tiered data centers to cloud services or to shared  centers. As a result, GSA had closed 118 data centers as of August  2018, including all of the agency’s tiered centers.

SSA developed an agency cloud initiative that encourages the  adoption of cloud technologies as part of the agency’s infrastructure  modernization. The agency reported that it is employing a hybrid  cloud strategy that is comprised of both private cloud and public cloud  services for the agency’s back office applications. By doing so, the  agency will consolidate and standardize SSA’s IT infrastructure  systems and software to simplify management of those resources and  reduce costs.

section_title: Emphasizing the Closure of Data Centers to Meet OMB Targets and Achieve Cost Savings
paragraphs: Three agencies (Agriculture, Justice, and EPA) reported that their  success in consolidation and optimization activities was due to focusing  on the closure of data centers. In doing so, they emphasized the  importance of closing data centers to reduce costs and achieve cost  savings and avoidances. For example,

Agriculture determined that the costs to improve DCOI performance  metrics in its agency-owned data centers were prohibitive.  Accordingly, the department decided that the only viable alternative  was to close data centers to remove underperforming centers and  improve optimization metrics performance and reduce costs. As a  result, Agriculture reported that it had closed 2,253 data centers  through August 2018. In addition, the department reported that it had  improved its security posture, reduced its real estate footprint, and  achieved realized cost savings and avoidance of $51.8 million from  fiscal year 2012 through 2018.

Justice reported that it took a practical approach to selecting the data  centers that would remain as its enterprise facilities, considering  factors such as the number of physical servers that could be  eliminated, the efficiency of the remaining hardware, and potential  labor savings. The department reported that it focused on retaining  more efficient data centers (e.g., those with more efficient use of  electricity or virtualization), rather than simply keeping its biggest  existing data centers. As a result, Justice has closed 84 of its 110  data centers and achieved more than $128 million in cost savings and  avoidances as of August 2018.

EPA identified geographical consolidation as its best approach to  meeting DCOI goals. Specifically, in its data center consolidation plan,  the agency stated that, for geographic areas where it had multiple  data centers, a single facility was identified into which data center IT  assets would be consolidated. Using this approach, EPA had closed  43 of its 83 data centers as of August 2018.

section_title: Increasing the Use of Virtualization to Optimize Data Centers
paragraphs: Three agencies (Commerce, EPA, and SSA) reported that their success  in consolidation and optimization activities also was due to focusing on  the increased use of virtualization to run more software on the same or  a reduced amount of servers. In doing so, the agencies expected to  reduce costs by avoiding the purchase of additional servers to meet  computing demands or eliminating unnecessary hardware and floor  space in their data centers. For example,

Commerce focused on moving systems from physical hardware to  virtual servers, as part of its component offices’ plans to update  technology and in cases where the systems did not require a specific  type of server. Using this approach, the department reported that it  had reduced the number of physical servers in its data centers, and  was working to improve server utilization. The department also cited  the ability to automatically increase or decrease computing capability  through virtualization, such as when NOAA handled the increased  traffic to its hurricane-related web pages during Hurricanes Irma and  Harvey in 2017.

EPA used the agency’s data center consolidation plan to implement  an agency-wide “physical-to-virtual” policy that required offices to  convert existing physical servers to virtual servers wherever possible.  The agency also defined server and software standards for virtualized  platforms.

SSA reported that the agency’s goal, using its “Virtual 1st” policy, was  to have failover capability within the data center, disaster recovery  capability for both data centers, and balanced load capacity between  data centers. The agency reported that it has continued to virtualize  not only servers but storage and network applications, as well. For  example, SSA stated that it has taken steps to virtualize as much  storage as possible and used similar techniques to reduce the  physical hardware footprint on the data center floor, as well as power,  cooling, and network bandwidth requirements.

section_title: Employing an Organization-wide Communications Plan to Facilitate Adoption of Consolidation and Optimization Activities
paragraphs: Two agencies (Justice and GSA) reported that their success in  consolidation and optimization activities was due to employing an  organization-wide communications plan. In doing so, the agencies  adopted a structured method for communicating with agency offices to  improve acceptance and adoption of consolidation and optimization  activities. This also facilitated conflict resolution. For example,  Justice reported that it prioritized communications related to its Data  Center Transformation Initiative and established an all-encompassing  approach to initiative-related communications. To help communicate  all related directives, strategies, plans, statuses, and  accomplishments, the department used a variety of methods that  included:  regular meetings to share information, a dedicated email box to provide easy communication for answers  or information, without the need to know specific individuals, an intranet web page that provided general information,  instructions, templates, decisions, status information, and  accomplishments related to the initiative; and email broadcasts on an as-needed basis.

GSA reported that it communicated and collaborated frequently with  business stakeholders to identify the best time frames to move  systems, stagger transfers to minimize impact, and determine which  systems could be virtualized. The agency indicated that these  important factors required continuous communication between system  owners, system administrators, and business leadership. As a result,  the agency experienced minimal staff resistance to change and a  commitment to reach a consensus on moving forward with the  agency’s consolidation efforts.

The aforementioned practices included elements of sound management  techniques, such as gathering leadership support for a project and  developing a communications plan to foster adoption of organizational  changes. The practices also included activities that aligned with the core  tenets of DCOI to consolidate inefficient infrastructure, optimize existing  facilities, and achieve cost savings. Further, these practices each proved  effective for multiple agencies and, while they were not the only practices  that could be effective, they represent concepts that could provide the  foundation for an effective data center consolidation and optimization  program.

section_title: Conclusions
paragraphs: Federal data center consolidation efforts have been underway since 2010  and OMB’s fiscal year 2018 targets provided clear and transparent goals  that helped define the tangible benefits that DCOI was expected to  provide. However, most agencies continue to report mixed progress  against those targets. Although agencies have taken action to close about  half of the data centers in their combined inventories, 11 agencies did not  plan to meet all of their closure targets.

Further, the data center closures were expected to drive cost savings and  avoidances and, to the agencies’ credit, the closures have led to more  than $2.37 billion in planned and achieved cost savings and avoidances  from fiscal years 2016 through 2018. However, five agencies did not plan  to meet their cost savings targets. Until agencies consolidate the data  centers required to meet their targets, as well as identify and report the  associated cost savings, they will be challenged to realize expected  efficiencies and the full benefits of DCOI will not be fully realized.

Similarly, although OMB first established optimization metrics in May  2014, agencies continue to report only limited progress against the  current performance targets. While two agencies do not have a basis to  report any progress as they do not own any data centers, only two  agencies reported that they planned to achieve all of DCOI’s fiscal year  2018 optimization targets. Ensuring the optimized performance of data  centers is a key component to meeting OMB’s DCOI-wide savings goal  and the 20 agencies that did not have plans to meet their targets call into  question whether DCOI will realize its full potential savings.

Although many agencies have struggled to meet their individual DCOI  targets, other agencies have successfully met OMB’s goals for data  center closures, savings, and optimization. Six such agencies that we  identified reported on the importance of gathering leadership support,  effective communication, and alignment with the core tenets of DCOI. Key  practices such as these can play an important role in helping agencies  better meet the overall goals and mission of DCOI.

section_title: Recommendations
paragraphs: We are making a total of 36 recommendations to 22 of the 24 agencies in  our review. Specifically:  The Secretary of Agriculture should take action to meet the data center  optimization metric targets established by OMB under DCOI.  (Recommendation 1)

The Secretary of Commerce should take action to meet the data center  closure targets established under DCOI by OMB. (Recommendation 2)

The Secretary of Defense should take action to meet the data center  closure targets established under DCOI by OMB. (Recommendation 3)

The Secretary of Defense should identify additional savings opportunities  to achieve the targets for data center-related cost savings established  under DCOI by OMB. (Recommendation 4)

The Secretary of Defense should take action to meet the data center  optimization metric targets established under DCOI by OMB.  (Recommendation 5)

The Secretary of Energy should take action to meet the data center  closure targets established under DCOI by OMB. (Recommendation 6)

The Secretary of Energy should take action to meet the data center  optimization metric targets established under DCOI by OMB.  (Recommendation 7)

The Secretary of the Department of Health and Human Services (HHS)  should take action to meet the data center closure targets established  under DCOI by OMB. (Recommendation 8)

The Secretary of HHS should take action to meet the data center  optimization metric targets established under DCOI by OMB.  (Recommendation 9)

The Secretary of DHS should take action to meet the data center closure  targets established under DCOI by OMB. (Recommendation 10)

The Secretary of DHS should take action to meet the data center  optimization metric targets established under DCOI by OMB.  (Recommendation 11)

The Secretary of Interior should take action to meet the data center  closure targets established under DCOI by OMB. (Recommendation 12)

The Secretary of Interior should take action to meet the data center- related cost savings established under DCOI by OMB. (Recommendation  13)

The Secretary of Interior should take action to meet the data center  optimization metric targets established under DCOI by OMB.  (Recommendation 14)

The Attorney General should take action to meet the data center  optimization metric targets established for Justice under DCOI by OMB.  (Recommendation 15)

The Secretary of the Department of Labor (Labor) should take action to  meet the data center optimization metric targets established under DCOI  by OMB. (Recommendation 16)

The Secretary of State should take action to meet the data center closure  targets established under DCOI by OMB. (Recommendation 17)

The Secretary of State should take action to meet the data center  optimization metric targets established under DCOI by OMB.  (Recommendation 18)

The Secretary of Transportation should take action to meet the data  center closure targets established under DCOI by OMB.  (Recommendation 19)

The Secretary of Transportation should take action to meet the data  center optimization metric targets established under DCOI by OMB.  (Recommendation 20)

The Secretary of Treasury should take action to meet the data center  optimization metric targets established under DCOI by OMB.  (Recommendation 21)

The Secretary of VA should take action to meet the data center closure  targets established under DCOI by OMB. (Recommendation 22)

The Secretary of VA should take action to meet the data center-related  cost savings established under DCOI by OMB. (Recommendation 23)

The Secretary of VA should take action to meet the data center  optimization metric targets established under DCOI by OMB.  (Recommendation 24)

The Administrator of EPA should take action to meet the data center  closure targets established under DCOI by OMB. (Recommendation 25)

The Administrator of EPA should take action to meet the data center  optimization metric targets established under DCOI by OMB.  (Recommendation 26)

The Administrator of GSA should take action to meet the data center  optimization metric targets established under DCOI by OMB.  (Recommendation 27)

The Administrator of the National Aeronautics and Space Administration  (NASA) should take action to meet the data center optimization metric  targets established under DCOI by OMB. (Recommendation 28)

The Director of NSF should take action to meet the data center  optimization metric targets established under DCOI by OMB.  (Recommendation 29)

The Chairman of NRC should take action to meet the data center  optimization metric targets established under DCOI by OMB.  (Recommendation 30)

The Director of OPM should take action to meet the data center-related  cost savings established under DCOI by OMB. (Recommendation 31)

The Director of OPM should take action to meet the data center  optimization metric targets established under DCOI by OMB.  (Recommendation 32)

The Administrator of the Small Business Administration (SBA) should  take action to meet the data center optimization metric targets established  under DCOI by OMB. (Recommendation 33)

The Commissioner of SSA should take action to meet the data center- related cost savings established under DCOI by OMB. (Recommendation  34)

The Commissioner of SSA should take action to meet the data center  optimization metric targets established under DCOI by OMB.  (Recommendation 35)

The Administrator of USAID should take action to meet the data center  closure targets established under DCOI by OMB. (Recommendation 36)

section_title: Agency Comments and Our Evaluation
paragraphs: We requested comments on a draft of this report from OMB and the 24  agencies that we reviewed. Of the 22 agencies to which we made  recommendations, 11 agencies agreed with our recommendations; three  agencies agreed with some portion, but not all of the recommendations;  one agency disagreed with our recommendations; and seven agencies  did not state whether they agreed or disagreed with the  recommendations. In addition, OMB and two agencies to which we did  not make recommendations stated that they had no comments. Further,  multiple agencies provided technical comments, which we have  incorporated, as appropriate.

The following 11 agencies agreed with our recommendations:  In written comments from Commerce, State, NASA, SBA, and SSA,  the agencies stated that they agreed with the recommendations and  indicated their intent to address them. State also provided technical  comments, which we have incorporated, as appropriate. The  agencies’ comments are reprinted in appendices II through VI.

In written comments, Energy agreed with our recommendations to  meet its data center closure and optimization metric targets, and  described actions that the department planned to take in order to  address the recommendations. Energy initially estimated that it would  complete these actions by March 1, 2019; however, the department  subsequently revised its estimated completion date to April 15, 2019.  Energy also provided technical comments, which we have  incorporated, as appropriate. Energy’s comments are reprinted in  appendix VII.

In written comments, VA agreed with our recommendations to meet  its data center closure, cost savings, and optimization metric targets.  In addition, the department requested that we close our  recommendation related to data center closures on the basis of its  planned actions to implement a new inventory data collection tool and  methodology to improve how the department collects data center  inventory information, and a positive trend in its data center closures.

The department estimated that its planned actions would be  completed in March 2019 and reported that, as of November 2018, it  had closed 78 data centers in fiscal year 2018, as compared with 24  in fiscal year 2017.

However, as noted earlier in this report, we found that VA did not plan  to meet the closure goal for either tiered or non-tiered data centers,  which was the basis for our recommendation. While we acknowledge  and encourage VA’s reported closure progress, the department still  has not met its DCOI closure goals, as we recommended. Further, VA  did not provide an update on the status of its planned actions in time  for us to address them in this report. As such, we maintain that this  recommendation is still appropriate.

In addition, VA referred to OMB’s proposed changes to DCOI  guidance when describing actions that it planned to take to meet the  department’s cost savings and optimization metrics targets. However,  OMB staff told us that the August 2016 DCOI guidance will remain in  effect until the revised DCOI guidance is formally issued. Once OMB’s  new DCOI guidance is finalized, we plan to assess agency progress  against any revised targets, and we will continue to monitor the  department’s efforts to address our recommendation. VA’s comments  are reprinted in appendix VIII.

We received emails from officials of Agriculture, Justice, Transportation,  and OPM which stated that these agencies agreed with the  recommendations we directed to them. In addition, three agencies  agreed with some portion, but not all of our recommendations directed to  them:  In written comments, Defense stated that it agreed with our  recommendation to meet its data center closure targets. However, the  department partially agreed with our two other recommendations: to  identify additional data center-related savings opportunities and to  meet OMB’s data center optimization metric targets.

In partially agreeing with our recommendation on data center savings,  Defense asserted that it had already identified significant cost savings  through activities such as the identification of system migration  candidates and the use of cloud services, among others. The  department further stated that, while it would continue to optimize its  data centers, the need for IT would continue to grow, and this growth  might ultimately lead to an increase in total data center costs, despite  overall per unit cost reductions.

However, the department’s planned savings of $205.46 million  represented only 11 percent of its $1.8 billion savings goal by the end  of fiscal year 2018 and, as such, this limited progress by the  department formed the basis for our recommendation. As discussed  in our report, OMB plans to revise DCOI guidance and work with  agencies to set agency-specific targets. According to OMB staff, until  the guidance is revised, the current guidance and its targets are still  applicable. For these reasons, we maintain that our recommendation  is still appropriate.

Further, in partially agreeing with our recommendation to meet  optimization metric targets, Defense stated that the department will  continue to drive towards the achievement of data center optimization  targets. It added, however, that it would not invest resources to  improve the efficiency of data centers planned for closure and that, as  a result, the composite view of Defense’s data center efficiency would  fall short of meeting OMB’s targets.

Our review found that Defense did not plan on meeting any of OMB’s  five data center optimization metric targets by the end of fiscal year  2018. This finding was the basis for our recommendation. We  acknowledge Defense’s position that investing resources into  optimizing data centers that are already planned for closure would not  be the best use of taxpayer dollars. We also noted in our report that  OMB had proposed revising its optimization metrics, and that any  such changes had not yet been finalized. Our recommendation is not  intended to imply that an agency should meet a particular version of  OMB targets but, rather, that the agency should meet any targets that  are established by OMB. This would include any future changes to  DCOI targets. Accordingly, we maintain that our recommendation is  still appropriate and will continue to monitor the department’s efforts to  address our recommendation. Defense’s comments are reprinted in  appendix IX.

In written comments, DHS stated that it agreed with our  recommendation to meet its data center closure targets and disagreed  with our recommendation to meet its data center optimization metric  targets. Specifically, the department noted that it had met its tiered  data center closure targets, and was reviewing the status of its  remaining open non-tiered data centers. The department added that it  expected to complete this activity by March 31, 2019. However, the  department did not provide an update on its efforts in time to be  included in this report.

While we encourage DHS’s continued efforts to close its remaining  non-tiered data centers, we note that the department’s letter cites an  inventory of 18 open non-tiered facilities, which differs significantly  from the 202 non-tiered centers counted in our draft report, and which  DHS officials confirmed in November 2018. According to the  department, this discrepancy is because OMB issued revised  inventory reporting requirements in November 2018, and these  revised requirements exempted certain types of facilities from DCOI  reporting and resulted in the lower number.

These changes in reporting requirements are similar to the proposed,  but not yet finalized, revisions to the DCOI policy that are discussed  earlier in this report. However, OMB staff told us that the August 2016  DCOI guidance will remain in effect until the revised DCOI guidance is  formally issued. Once OMB’s new DCOI guidance is finalized, we plan  to assess agency progress against any revised targets, and we will  continue to monitor the department’s efforts to address our  recommendation.

Further, in disagreeing with our recommendation on meeting  optimization metrics, the department stated that, while the  recommendation was applicable under the original DCOI guidance  that OMB issued in August 2016, OMB’s proposed changes to DCOI  guidance would exempt most, if not all, DHS agency-owned data  centers from the optimization metrics. Consequently, the department  requested that our recommendation be closed.

In our review, we found that the department did not plan on meeting  any of OMB’s five data center optimization metric targets established  under DCOI. This finding was the basis for our recommendation on  meeting optimization metrics. Also, while OMB has proposed changes  to its metrics, as we noted previously, it has not provided a date for  when any such proposed changes will be finalized and implemented;  and, according to OMB staff, until the changes to DCOI guidance are  finalized, the current guidance is still applicable. Further, our  recommendations do not specify that an agency should meet any  particular version of OMB targets, but rather, that an agency should  meet the targets established by OMB. This would include any future  changes to DCOI targets. Accordingly, we maintain that our  recommendation is still appropriate. DHS also provided technical  comments, which we have incorporated, as appropriate. DHS’s  comments are reprinted in appendix X.

In written comments, Interior stated that it partially agreed with our  recommendation to meet its data center closure targets and disagreed  with our two recommendations to meet its data center-related cost  savings target and its data center optimization metric targets. For all  three recommendations, the department stated that OMB had  proposed changes to DCOI guidance that would result in new targets  for closures, cost savings, and optimization metrics and that Interior  planned to adopt the new policy and work through OMB to establish  its new targets.

As noted in our report, Interior met its target for tiered data center  closures, but did not plan to meet the closure goal for non-tiered data  centers. Further, the department planned on achieving only $15.95  million of its $88.19 million savings target (18 percent) by the end of  fiscal year 2018, and did not plan on meeting any of OMB’s five data  center optimization metric targets. These three findings were the basis  for our recommendations to the department.

We also noted that, as part of OMB’s proposed changes to DCOI  guidance, it planned to work with agencies to set agency-specific  targets for data center closures and planned to modify the metrics  currently used by agencies to monitor the performance of their data  centers. However, as previously mentioned, OMB has not provided a  date for when these proposed changes will be finalized and  implemented and, according to OMB staff, until the changes to DCOI  guidance are finalized, the 2016 guidance is still applicable.  Furthermore, our recommendations do not specify that an agency  should meet any particular version of OMB targets, but should meet  any targets that are established by OMB. This would include any  future changes to DCOI targets. As such, we maintain that our  recommendations are appropriate. Interior’s comments are reprinted  in appendix XI.

One agency disagreed with all of our recommendations:  In written comments, HHS disagreed with our two recommendations  to meet its data center closure targets and data center optimization  metric targets. In regard to both recommendations, the department  disagreed with being held to what it termed “expired requirements”  from DCOI guidance, pending the assignment of new targets being  established by OMB.

As noted in our report, HHS met its target for tiered data center  closures, but did not plan to meet the closure target for non-tiered  data centers. We also found that HHS did not meet any of OMB’s five  optimization metric targets and had planned to meet only one of the  five by end of fiscal year 2018. These findings were the basis for the  two recommendations that we made to the department.

We also noted that, as part of OMB’s proposed changes to DCOI  guidance, OMB planned to work with agencies to set agency-specific  targets for data center closures and planned to modify the metrics  currently used by agencies to monitor the performance of their data  centers. However, as previously mentioned, OMB did not provide a  date for when these proposed changes will be finalized and  implemented and, according to its staff, until the changes to DCOI  guidance are finalized, the current guidance is still applicable. Further,  our recommendations do not specify that an agency should meet any  particular version of OMB targets, but rather, that the agency should  meet the targets established by OMB. This would include any future  changes to DCOI targets. Accordingly, we maintain that our  recommendations are still appropriate. HHS’s comments are reprinted  in appendix XII.

Further, seven agencies did not agree or disagree with the  recommendations:  In written comments, EPA did not state whether it agreed or disagreed  with our recommendations to meet its data center closure and data  center optimization metrics targets. However, the agency requested  that we close our recommendations, citing its reported progress in  closing 21 of 34 targeted data centers and OMB’s proposed changes  in its draft DCOI guidance that could result in revised closure targets  and optimization metrics.

As stated in our report, we found that EPA did not plan to meet its  closure target for tiered or non-tiered data centers, nor did it plan to  meet its data center optimization targets; these findings were the  basis for our recommendations. We also noted that, as part of OMB’s  proposed changes to DCOI, OMB planned to work with agencies to  set agency-specific targets for data center closures and planned to  modify the metrics currently used by agencies to monitor the  performance of their data centers. However, OMB has not provided a  date for when these proposed changes will be finalized and  implemented and, according to OMB staff, until the changes to DCOI  guidance are finalized, the current guidance is still applicable. Further,  our recommendations do not specify that an agency should meet any  particular version of OMB targets, but that it should meet the targets  established by OMB. This would include any future changes to DCOI  targets. Accordingly, we maintain that our recommendations are  appropriate and should remain open. EPA also provided technical  comments, which we have incorporated, as appropriate. The agency’s  comments are reprinted in appendix XIII.

In written comments, GSA did not state whether it agreed or  disagreed with our recommendation to meet the agency’s data center  optimization metrics targets. Specifically, the agency stated that it had  complied with revised inventory reporting requirements, which OMB  provided to agencies in November 2018 and which eliminated non- tiered data centers from the requirement to meet optimization targets.  As a result, the agency noted that it no longer had a basis to measure  and report on the one metric our report cited as applicable to GSA  (i.e., server utilization and automated monitoring) and asked that we  withdraw the recommendation.

These changes in reporting requirements are similar to the proposed,  but not yet finalized, revisions to the DCOI policy that are discussed  earlier in the report. However, OMB staff told us that the August 2016  DCOI guidance is still in effect until the revised DCOI guidance is  formally issued. Until OMB’s new DCOI guidance is finalized and  agency progress against any revised targets can be evaluated, we  maintain that our recommendation to meet the agency’s optimization  metrics targets is appropriate, and we will continue to monitor the  agency’s efforts to address it. GSA’s comments are reprinted in  appendix XIV.

In written comments, NSF did not state whether it agreed or disagreed  with our recommendation. The agency’s comments are reprinted in  appendix XV.

In written comments, NRC agreed with the draft report, but did not  state whether it agreed or disagreed with our recommendation. The  agency’s comments are reprinted in appendix XVI.

In written comments USAID did not state whether it agreed or  disagreed with the draft report’s recommendation but agreed with our  finding that the agency no longer had any tiered data centers.  However, USAID stated that it had met DCOI’s closure targets for the  agency by closing its 4 non-tiered data centers, and requested that we  close our recommendation to meet those targets.

While we encourage USAID’s continued efforts to close its remaining  non-tiered data centers, we note that the agency’s letter cites an  inventory of 4 non-tiered facilities, which differs significantly from the  83 non-tiered centers counted in our draft report, and which USAID  officials confirmed in October 2018. As USAID communicated in  subsequent emails, this discrepancy is because OMB issued revised  inventory reporting requirements in November 2018 and these revised  requirements exempted certain types of facilities from DCOI reporting,  which resulted in the lower number.

These changes in reporting requirements are similar to the proposed,  but not yet finalized, revisions to the DCOI policy that are discussed  earlier in the report. However, OMB staff told us that the August 2016  DCOI guidance is still in effect until the revised DCOI guidance is  formally issued. Until OMB’s new DCOI guidance is finalized and  agency progress against any revised targets can be evaluated, we  maintain that our data center closure recommendation is appropriate,  and we will continue to monitor the agency’s efforts to address it. The  agency’s comments are reprinted in appendix XVII.

In emails received from Labor’s GAO liaison in the department’s  Office of the Assistant Secretary for Policy on January 8, 2019, and  from an audit liaison in Treasury’s Office of the CIO on February 1,  2019, both departments did not state whether they agreed or  disagreed with our respective recommendations.

Finally, in emails received from a Management and Program Analyst in  Education’s Office of the Secretary/Executive Secretariat on January 8,  2019; an audit liaison in HUD’s Office of the CIO, Audit Compliance  Branch on February 15, 2019; and a GAO liaison in OMB’s Office of  General Counsel on February 25, 2019, these agencies stated that they  had no comments on the draft report.

We are sending copies of this report to interested congressional  committees, the Director of OMB, the secretaries and heads of the  departments and agencies addressed in this report, and other interested  parties. In addition, the report will be available at no charge on GAO’s  website at http://www.gao.gov.

If you or your staffs have any questions about this report, please contact  me at (202) 512-4456 or harriscc@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff who made major contributions to this report  are listed in appendix XVIII.

section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives for this engagement were to (1) determine agencies’  progress in data center closures and achievement in related savings to  date and describe plans for future savings, (2) evaluate the agencies’  progress against OMB’s data center optimization targets, and (3) identify  effective agency practices for achieving data center closures, cost  savings, and optimization.

To address the first objective, for data center closures, we obtained and  analyzed August 2018 data center inventory documentation from the 24  departments and agencies (agencies) that participate in OMB’s Data  Center Optimization Initiative (DCOI). To determine data center closures  to date, we totaled their reported closures from fiscal year 2010 through  August 2018 and to identify future closures, we totaled their reported  planned closures through fiscal year 2018. We also compared agencies’  completed and planned closures to OMB’s fiscal year 2018 consolidation  goals, as documented in its August 2016 memorandum (M-16-19).

To verify the quality, completeness, and reliability of each agency’s data  center inventory, we compared information on completed and planned  data center closures to similar information reported on OMB’s IT  Dashboard—a public website that provides information on federal  agencies’ major IT investments. We also checked for missing data and  other errors, such as missing closure status information. In some cases  identified, we followed-up with agency officials to obtain further  information. We determined that the data were sufficiently complete and  reliable to report on agencies’ consolidation progress and planned  closures.

For cost savings and avoidance we obtained and analyzed  documentation from the 24 DCOI agencies. This documentation is  required by OMB’s March 2013 and August 2016 memorandums and  included the agencies’ quarterly reports of cost savings and avoidances  posted to their digital services websites and their DCOI strategic plans.  To determine cost savings achieved, we totaled agencies’ reported  savings and avoidances from the start of fiscal years 2012 through  August 2018, as found in the August 2018 quarterly reports posted to the  agencies’ digital services websites. To identify future planned savings,  we totaled the agencies’ projected savings and avoidances from fiscal  years 2016 through 2018, as reported in their DCOI strategic plans.

To assess the quality, completeness, and reliability of each agency’s data  center consolidation cost savings information, we used the latest version  of each agency’s quarterly cost savings report and DCOI strategic plan,  as of August 2018. We also reviewed the quarterly reports and DCOI  strategic plans for missing data and other errors, such as missing cost- savings information. In addition, we compared agencies cost savings and  avoidances with data from our most recent data center consolidation  report. As a result, we determined that the data were sufficiently  complete and reliable to report on agencies data center consolidation  cost-savings information.

For our second objective, we analyzed the August 2018 data center  optimization progress information of the 24 DCOI agencies. This progress  information was obtained from the IT Dashboard—an OMB public website  that provides information on federal agencies’ major IT investments. To  assess agencies’ planned optimization progress, we obtained the planned  optimization performance from the 22 agencies’ DCOI strategic plans. We  then compared the agencies’ current and planned optimization progress  information to OMB’s fiscal year 2018 optimization targets, as  documented in its August 2016 memorandum. Although OMB’s  memorandum establishes a single optimization target value for the server  utilization and automated monitoring metric, the IT Dashboard displays  agencies’ progress for tiered and non-tiered data centers separately. To  report consistently with OMB’s implementation memorandum, we  combined the progress information for tiered and non-tiered data centers  into a single assessment in this report.

In addition, to assess the reliability of the planned optimization milestones  in the DCOI strategic plans, we reviewed agencies’ documentation to  identify any missing or erroneous data. We also compared the planned  data center optimization milestones contained in agencies’ documentation  against current optimization progress information obtained from the IT  Dashboard; we then discussed any discrepancies or potential errors that  we identified with agency officials to determine the causes or request  additional information. As a result of these efforts, we were able to  determine whether each agency’s strategic plan information was  sufficiently reliable for reporting on plans to meet or not meet OMB’s fiscal  year 2018 optimization targets.

To assess the reliability of agencies’ optimization progress information on  OMB’s IT Dashboard, we reviewed the information for errors or missing  data, such as progress information that was not available for certain  metrics. We also compared agencies’ optimization progress information  across multiple reporting quarters to identify any inconsistencies in  agencies’ reported progress. We discussed with staff from OMB’s Office  of the Federal Chief Information Officer any discrepancies or potential  errors identified to determine the causes.

To identify effective agency practices for achieving data center closures,  cost savings, and optimization progress, we selected two of the highest  performing departments or agencies for each of those three data center  areas that we reported on in our May 2018 report. For the data center  inventory closures area, we selected the Departments of Agriculture  (Agriculture) and Justice (Justice) from among the five agencies that had, as of August 2017, reached or exceeded both their tiered and non-tiered  data center closure targets for the end of fiscal year 2018. For the cost  savings area, we identified two departments and two small agencies  reporting the highest cost savings DCOI to date, as of August 2017. From  those, we selected one department (Commerce) and one small agency  (the General Services Administration) to provide balance relative to  agency size. For effective practices related to optimization performance,  we reviewed agencies’ reported optimization performance as of August  2017 and selected the two highest-performing agencies in this area (the  Social Security Administration and the Environmental Protection Agency),  since they were the only two agencies reporting that they met more than  half of OMB’s optimization targets. Selecting these agencies was  designed to provide anecdotal information that could assist agencies  struggling with DCOI implementation. The examples they provided are  not findings nor should they be taken to be representative of all the  agencies participating in DCOI.

We asked each selected agency to identify practices that they found  effective in implementing DCOI at their agency and in meeting OMB’s  established targets in each of the areas, not just the area for which they  were selected. We also solicited examples that demonstrated how those  practices helped agency implementation or the benefits from  implementing DCOI. Additionally, we considered information and  examples that these agencies provided as part of our work to identify  FITARA best practices. We analyzed the responses to determine the  practices and reported those that were identified by at least two agencies.

We conducted this performance audit from April 2018 to April 2019 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

section_title: Appendix II: Comments from the Department of Commerce

section_title: Appendix III: Comments from the Department of State

section_title: Appendix IV: Comments from the National Aeronautics and Space Administration

section_title: Appendix V: Comments from the Small Business Administration

section_title: Appendix VI: Comments from the Social Security Administration

section_title: Appendix VII: Comments from the Department of Energy

section_title: Appendix VIII: Comments from the Department of Veterans Affairs

section_title: Appendix IX: Comments from the Department of Defense

section_title: Appendix X: Comments from the Department of Homeland Security

section_title: Appendix XI: Comments from the Department of the Interior

section_title: Appendix XII: Comments from the Department of Health and Human Services

section_title: Appendix XIII: Comments from the Environmental Protection Agency

section_title: Appendix XIV: Comments from the General Services Administration

section_title: Appendix XV: Comments from the National Science Foundation

section_title: Appendix XVI: Comments from the Nuclear Regulatory Commission

section_title: Appendix XVII: Comments from the U.S. Agency for International Development

section_title: Appendix XVIII: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact

section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, individuals making contributions  to this report included Dave Powner (director), Dave Hinchman (assistant  director), Justin Booth (analyst-in-charge), Alexander Bennett, Chris  Businsky, Nancy Glover, and Jonathan Wall.

Fast facts: Federal agencies operate thousands of data centers and since 2010 have been required to close unneeded facilities and improve the performance of the remaining centers. Across the government, agencies have closed 6,250 centers to date and saved $2.7 billion.

However, only 2 agencies in our review planned to meet September 2018 government-wide optimization goals that include, for example, a target for how much time data servers sit unused.

We recommended that agencies improve data centers' operational efficiency and identify further savings.

Question-summary annotations: questions: question: What did the 24 agencies participating in DCOI report?
answer: The 24 agencies participating in the Office of Management and Budget's (OMB) Data Center Optimization Initiative (DCOI) reported mixed progress toward achieving OMB's goals for closing data centers and realizing the associated savings by September 2018.
child_questions: question: What specifically did each of the agencies report?
answer: As of August 2018, 13 agencies reported that they had met, or had plans to meet, all of their OMB-assigned closure goals by the deadline. However, 11 agencies reported that they did not have plans to meet their goals. Further, 16 agencies reported that, as of August 2018, they had met, or planned to meet, their cost savings targets, for a total of $2.36 billion in cost savings for fiscal years 2016 through 2018. Three agencies did not have a cost savings target and did not report any achieved savings.
child_questions: question: What is the problem with 16 of the agencies' cost savings target?
answer: Further, 16 agencies reported that, as of August 2018, they had met, or planned to meet, their cost savings targets, for a total of $2.36 billion in cost savings for fiscal years 2016 through 2018. This is about $0.38 billion less than OMB's DCOI savings goal of $2.7 billion.

question: Why is there a shortfall from the DCOI savings goal?
answer: This shortfall is the result of 5 agencies reporting less in planned cost savings and avoidances in their DCOI strategic plans, as compared to their savings targets established for them by OMB.

questions: question: What did the 24 agencies report?
answer: In addition, the 24 agencies reported limited progress against OMB's five data center optimization targets for server utilization and automated monitoring, energy metering, power usage effectiveness, facility utilization, and virtualization.
child_questions: question: What specifically did the agencies report regarding their targets?
answer: As of August 2018, the agencies reported that 3 had met three targets, 9 had met one target, and 10 met none of the targets. Two agencies did not have a basis to report on progress as they do not own any data centers.
child_questions: question: Why did two agencies not have  a basis to report progress on?
answer: Two agencies did not have a basis to report on progress as they do not own any data centers.

question: What did the agencies report regarding the OMB's optimization goals?
answer: Further, as of August 2018, 20 agencies did not plan to meet all of OMB's fiscal year 2018 optimization goals. Specifically, only 2 agencies reported plans to meet all applicable targets; 6 reported that they did not plan to meet any of the targets (see figure).

questions: question: What did successful agencies report?
answer: We selected 6 agencies that had demonstrated success towards meeting their DCOI goals and those agencies reported a number of key practices that contributed to their efforts.
child_questions: question: What was important to these agencies' successes?
answer: The officials noted the importance of, among other things, obtaining executive leadership support for consolidation and optimization activities, employing an organization-wide communications plan, and focusing on data center closures. The officials also cited the use of past experience and lessons learned to inform improvements to future consolidation plans and processes.
child_questions: question: Why was past experience important for success?
answer: The officials also cited the use of past experience and lessons learned to inform improvements to future consolidation plans and processes.

questions: question: What did 2014 legislation allow GAO to do?
answer: The 2014 legislation included a provision for GAO to annually review agencies' data center inventories and strategies.
child_questions: question: What were GAO's objectives in 2014 legislature?
answer: Accordingly, GAO's objectives were to (1) evaluate agencies' progress and plans for data center closures and cost savings; (2) assess agencies' progress against OMB's data center optimization targets; (3) and identify effective agency practices for achieving data center closures, cost savings, and optimization progress.
child_questions: question: How did GAO complete these objectives?
answer: To do so, GAO assessed the 24 DCOI agencies' data center inventories as of August 2018; reviewed their reported cost savings documentation; evaluated their data center optimization strategic plans; and assessed their progress against OMB's established optimization targets.

question: What did GAO solicit?
answer: GAO also solicited practices that selected agencies reported to be effective in meeting DCOI goals.