Data Center Optimization: Additional Agency Actions Needed to Meet OMB Goals
Released: 2019-04-11T00:00:00

Letter

Background

According to OMB, federal agencies reported that they operated 432 data centers in 1998, 2,094 in July 2010, and 9,995 in August 2016. Operating such a large number of centers has been, and continues to be, a significant cost to the agencies. For example, in 2007, EPA estimated that the annual cost for electricity to operate federal servers and data centers across the government was about $450 million.

Further, according to the Department of Energy (Energy), a typical government data center has 100 to 200 times the energy use intensity of a commercial building. However, in 2009, OMB reported server utilization rates as low as 5 percent across the federal government’s estimated 150,000 servers. All of these factors contributed to OMB recognizing the need to establish a coordinated, government-wide effort to improve the efficiency, performance, and environmental footprint of federal data center activities.

OMB and the Federal CIO Established FDCCI

Concerned about the size of the federal data center inventory and the potential to improve the efficiency, performance, and environmental footprint of federal data center activities, OMB’s Federal CIO established FDCCI in February 2010. This initiative’s four high-level goals were to reduce the overall energy and real estate footprint of government data centers; reduce the cost of data center hardware, software, and operations; increase the overall IT security posture of the government; and shift IT investments to more efficient computing platforms and technologies.

In February 2010, OMB required all of the agencies participating in the FDCCI to submit a data center inventory and a consolidation plan. In October 2010, OMB also clarified the definition of a data center and noted that, for the purposes of FDCCI, a data center was to be defined as any room used for the purpose of processing or storing data that is larger than 500 square feet and meets stringent availability requirements. Under this definition, OMB reported that agencies had identified 2,094 data centers as of July 2010.

However, in 2011, the Federal CIO expanded the definition to include a facility of any size and OMB published its revised definition in March 2012. Based on the revised definition, OMB estimated that there were a total of 3,133 federal data centers in December 2011. In addition, its goal was to consolidate approximately 40 percent, or 1,253 of these data centers, for a savings of approximately $3 billion by the end of 2015.

Figure 1 shows data center server racks at SSA’s National Support Center in 2017.

The number of federal data centers reported by agencies has continued to grow since 2011. In May 2018, we reported that agencies had collectively identified a total of 12,062 data centers in their inventories as of August 2017—an increase of about 9,000 data centers compared to OMB’s October 2011 estimate. According to the Federal CIO, the increase in the number of data centers was primarily due to the expanded definition of a data center (discussed later in this report) and improved inventory reporting by the agencies. See figure 2 for a depiction of the increase in the number of data centers from 1998 through August 2018.

Further, OMB placed greater emphasis on data center optimization to improve the efficiency of federal data centers when it issued memorandum M-13-09 in March 2013. Specifically, OMB stated that, to more effectively measure the efficiency of an agency’s data center assets, agencies would also be measured by the extent to which their primary data centers are optimized for total cost of ownership by incorporating metrics for data center energy, facility, labor, and storage, among other things.

Subsequently, in May 2014, OMB issued memorandum M-14-08, which established a set of data center optimization metrics to measure agency progress, along with target values for each metric. All agencies were expected to achieve the target values by the end of fiscal year 2015.

IT Acquisition Reform Law Enhanced Data Center Consolidation and Optimization Efforts

Recognizing the importance of reforming the government-wide management of IT, Congress enacted FITARA in December 2014. Among other things, the law required agencies to:

Submit to OMB a comprehensive inventory of the data centers owned, operated, or maintained by or on behalf of the agency.

Submit, by the end of fiscal year 2016, a multi-year strategy to achieve the consolidation and optimization of the agency’s data centers. The strategy was to include performance metrics that were consistent with the government-wide data center consolidation and optimization metrics.

Report progress toward meeting government-wide data center consolidation and optimization metrics on a quarterly basis to OMB’s Administrator of the Office of Electronic Government.

In addition, according to FITARA, the Office of Electronic Government at OMB was to:

Establish metrics applicable to the consolidation and optimization of data centers (including server efficiency), ensure that agencies’ progress toward meeting government-wide data center consolidation and optimization metrics is made publicly available, review agencies’ inventories and strategies to determine whether they are comprehensive and complete, and monitor the implementation of each agency’s strategy.

Develop and make publicly available not later than December 19, 2015, a goal broken down by year for the amount of planned cost savings and optimization improvements to be achieved through FDCCI and, for each year thereafter until October 1, 2020, compare reported cost savings and optimization improvements against those goals.

OMB Established DCOI to Address FITARA Data Center Provisions

OMB issued memorandum M-16-19 in August 2016 to establish DCOI and included guidance on how to implement the data center consolidation and optimization provisions of FITARA. Among other things, the guidance required agencies to consolidate inefficient infrastructure, optimize existing facilities, improve their security posture, and achieve cost savings. For example, each agency was required to maintain a complete inventory of all data center facilities owned, operated, or maintained by or on its behalf, and measure progress toward defined optimization performance metrics on a quarterly basis as part of its data center inventory submission.

OMB’s memorandum also directed each agency to develop a DCOI strategic plan that defined its data center strategy for fiscal years 2016 through 2018. Among other things, this strategy was to include a timeline for agency consolidation and optimization activities, with an emphasis on cost savings and optimization performance benchmarks that the agency could achieve between fiscal years 2016 and 2018. For example, each agency was required to develop cost savings targets due to consolidation and optimization actions and report any realized cost savings. OMB required each agency to publicly post its DCOI strategic plan to its agency-owned digital strategy website by September 30, 2016, and to post subsequent strategic plan updates by April 14, 2017 and April 13, 2018.

Further, the memorandum stated that OMB was to maintain a public dashboard (referred to as the IT Dashboard) to display government-wide and agency-specific progress in areas such as planned and achieved data center closures, consolidation-related cost savings, and data center optimization performance information. In this regard, OMB began including data center consolidation and optimization progress information on the IT Dashboard in August 2016.

OMB’s memorandum also provided new guidance for the classification of a physical data center and expanded the definition of a data center. According to the revised definition, a room with at least one server that provides services (whether in a production, test, staging, development, or any other environment) should be considered a data center, while a room containing only print servers, routing equipment, switches, security devices (such as firewalls), or other telecommunication components, was not to be considered a data center.

In light of this new definition, OMB directed each agency to perform a comprehensive review of its data centers and maintain a complete and updated data center inventory. Further, OMB directed each agency to categorize each of its data centers as either a tiered data center or a non- tiered data center. OMB’s memorandum defined a tiered data center as one that uses each of the following: a separate physical space for IT infrastructure; an uninterruptible power supply; a dedicated cooling system or zone; and a backup power generator for a prolonged power outage.

According to the memorandum, all other data centers were to be considered non-tiered.

Moreover, OMB guidance included a series of performance metrics in the areas of data center closures, cost savings, and optimization progress.

Data center closures: According to the guidance, agencies were to close at least 25 percent of tiered data centers government-wide, excluding those approved as inter-agency shared services providers, by the end of fiscal year 2018. Further, agencies were to close at least 60 percent of non-tiered data centers government-wide by the end of fiscal year 2018. OMB’s guidance further notes that, in the long term, all agencies should continually strive to close all non-tiered data centers, noting that server rooms and closets pose security risks and management challenges and are an inefficient use of resources.

Cost savings: According to the guidance, agencies were to reduce government-wide annual costs attributable to physical data centers by at least 25 percent, resulting in savings of at least $2.7 billion for fiscal years 2016 through 2018.

Data center optimization: According to the guidance, agencies were to measure progress against a series of new data center performance metrics in the areas of server utilization, energy metering, power usage, facility utilization, and virtualization. Further, OMB’s guidance established target values for each metric that agencies were to achieve by the end of fiscal year 2018.

OMB’s guidance further noted that agency progress against these performance metrics was to be measured by OMB on a quarterly basis, using agencies’ data center inventory submissions and OMB-defined closures, cost savings, and optimization targets.

OMB Published Proposed Changes to DCOI in November 2018

In November 2018, OMB published proposed changes to DCOI for public comment. The changes focus federal consolidation and optimization efforts on agencies’ larger, tiered data centers and also de-emphasize the consolidation of non-tiered facilities and other smaller spaces. The draft guidance also revises the classification of data centers and data center optimization metrics.

The draft guidance redefines a data center as a purpose-built, physically separate, dedicated space that meets certain criteria. Similarly, OMB does not plan to continue to report on spaces not designed to be data centers. According to the draft, OMB also plans to work with agencies to set agency-specific goals for data center closures and cost savings and to update these targets from those set in OMB’s August 2016 memorandum to match agencies’ current status and progress.

Additionally, the proposed changes to DCOI make several changes to the metrics currently used by agencies to monitor the performance of their data centers. Specifically, of the five metrics currently in use (described in detail later in this report), OMB proposes updating three, removing two, and adding one new metric.

The draft guidance states that public comments will be collected through the end of December 2018, but does not provide a date for when the proposed changes will be finalized and implemented. However, the draft does state that the new guidance will sunset on September 30, 2020, a date that coincides with the extension of FITARA’s data center provisions.

Agencies Have Taken Limited Action to Address Prior GAO Recommendations

Since the enactment of FITARA in December 2014, we have reviewed and verified annually for the quality and completeness of each agency’s (covered by the law) inventory and DCOI strategy. We have also published reports documenting the findings from each of our reviews. In addition, we have examined and reported on agencies’ efforts to optimize their data centers, as well as the challenges encountered and successes achieved.

In a report that we issued in March 2016, we noted that agencies had reported significant data center closures—totaling more than 3,100 through fiscal year 2015—with the Departments of Agriculture, Defense (Defense), the Interior (Interior), and the Treasury (Treasury) accounting for 84 percent of the total. Although the agencies fell short of OMB’s fiscal year 2015 consolidation goal, their plans identified about 2,100 additional centers planned for closure through fiscal year 2019.

Agencies also reported significant consolidation cost savings and avoidances—totaling about $2.8 billion through fiscal year 2015 and expected to increase to over $8.0 billion in future years. The Departments of Commerce, Defense, Homeland Security (DHS), Transportation (Transportation), and the Treasury accounted for 96 percent of the total planned savings.

However, we pointed out that many agencies lacked complete cost savings goals for the next several years despite having closures planned. In addition, we reported that 22 agencies had made limited progress against OMB’s fiscal year 2015 data center optimization performance metrics, such as the utilization of data center facilities. Accordingly, we recommended that the agencies take actions to complete their cost savings targets and improve optimization progress. As of December 2018, 18 of the 32 recommendations from this report had yet to be fully addressed.

In May 2017, we reported that the agencies were reporting significant data center closures—totaling more than 4,300 through August 2016— with Agriculture, Defense, Interior, and the Treasury accounting for 84 percent of the total. The agencies’ plans for 2016 had identified more than 1,200 additional centers for closure through fiscal year 2019.

Agencies also reported significant consolidation and optimization cost savings and avoidances, which totaled about $2.3 billion through August 2016. However, reductions in the amount of achieved savings reported to OMB, particularly by the Treasury, resulted in a net decrease of more than $400 million in these savings, compared to amounts we previously reported as planned in 2015.

Further, our report noted that, as of December 2016, agencies’ total planned cost savings of about $656 million were more than $3.3 billion less compared to the amounts that we reported in 2015, and more than $2 billion less than OMB’s fiscal year 2018 cost savings goal of $2.7 billion. This reduction in planned savings was the result of eight agencies reporting less in planned cost savings and avoidances in their DCOI strategic plans compared to the savings amounts previously reported to us in November 2015. The reduction also reflected the absence of cost savings information for one agency (Defense) that did not submit its strategic plan in time for our review.

In addition, our May 2017 report identified weaknesses in agencies’ DCOI strategic plans. Of the 23 agencies that had submitted their strategic plans at the time of our review, 7 agencies—Agriculture, the Department of Education (Education), DHS, and the Department of Housing and Urban Development (HUD); GSA; the National Science Foundation (NSF); and the Office of Personnel Management (OPM)—had addressed all five required elements of a strategic plan, as identified by OMB (such as providing information related to data center closures and cost savings metrics). The remaining 16 agencies that submitted their plans either partially met or did not meet the requirements. We also pointed out that there were inconsistencies in the reporting of cost savings in the strategic plans of 11 agencies.

Given these findings, we recommended that OMB improve its oversight of agencies’ DCOI strategic plans and their reporting of cost savings and avoidances. We also recommended that 16 agencies and Defense (which did not submit a plan in time for our review) complete the missing elements in their strategic plans, and that 11 agencies ensure the reporting of consistent cost savings and avoidance information to OMB. As of December 2018, 10 of the 30 recommendations had not been fully addressed.

In a subsequent report that we issued in August 2017, we noted that 22 of the 24 agencies required to participate in the OMB DCOI had reported (collectively) limited progress against OMB’s fiscal year 2018 performance targets for the five optimization metrics. The 2 remaining agencies, Education and HUD, did not own any data centers and, therefore, did not have a basis to report on progress. Specifically, for each of the five targets, no more than 5 agencies reported that they had met or exceeded that specific target. We reported that this limited progress against OMB’s optimization targets was due, in part, to agencies not fully addressing our prior recommendations in this area.

In addition, we noted in the report that most agencies had not yet implemented automated monitoring tools to measure server utilization, as required by the end of fiscal year 2018. Specifically, 4 agencies reported that they had fully implemented such tools, 18 reported that they had not yet done so, and 2 did not have a basis to report on progress because they did not own any data centers. Accordingly, we recommended that OMB require that agencies include plans, as part of existing OMB reporting mechanisms, to implement automated monitoring tools at their agency-owned data centers. We also recommended that the 18 agencies that did not have fully documented plans take action, within existing OMB reporting mechanisms, to complete plans describing how they intended to achieve OMB’s requirement to implement automated monitoring tools at all agency-owned data centers by the end of fiscal year 2018. As of December 2018, none of our 19 recommendations had been fully addressed.

Most recently, in May 2018, we noted that the 24 agencies participating in DCOI had reported mixed progress toward achieving OMB’s goals for closing their data centers by September 2018. Thirteen agencies reported that they had either already met, or planned to meet, all of their OMB- assigned goals by the deadline. However, 4 agencies reported that they did not have plans to meet all of their assigned goals and 2 agencies were working with OMB to establish revised targets.

With regard to agencies’ progress in achieving cost savings, 20 agencies reported, as of August 2017, that they had achieved $1.04 billion in cost savings for fiscal years 2016 and 2017. In addition, the agencies’ DCOI strategic plans identified an additional $0.58 billion in planned savings— for a total of $1.62 billion for fiscal years 2016 through 2018. This total was approximately $1.12 billion less than OMB’s DCOI savings goal of $2.7 billion. This shortfall was the result of 12 agencies reporting less in planned cost savings and avoidances in their DCOI strategic plans, as compared to the savings targets established for them by OMB.

In addition, the 24 agencies reported limited progress against OMB’s five data center optimization targets for server utilization and automated monitoring, energy metering, power usage effectiveness, facility utilization, and virtualization. As of August 2017, 1 agency had met four targets, 1 agency had met three targets, 6 agencies had met either one or two targets, and 14 agencies reported meeting none of the targets. Further, as of August 2017, most agencies were not planning to meet OMB’s fiscal year 2018 optimization targets. Specifically, 4 agencies reported plans to meet all of their applicable targets by the end of fiscal year 2018; 14 reported plans to meet some of the targets; and 4 reported that they did not plan to meet any targets.

Because GAO had made a number of recommendations to OMB and the 24 DCOI agencies to help improve the reporting of data center-related cost savings and to achieve optimization targets, we did not make new recommendations and noted that, as of March 2018, 74 of the 81 prior recommendations had not been fully addressed. While agencies have made considerable progress, as of December 2018, 47 of the 81 recommendations had not been fully addressed.

Agencies Reported Mixed Results in Efforts and Plans to Meet OMB’s Targets for Data Center Closures and Cost Savings

According to OMB guidance, agencies were expected to close at least 25 percent of tiered data centers government-wide, by the end of fiscal year 2018. In addition, agencies were to close at least 60 percent of non-tiered data centers government-wide by this same deadline. Further, agencies were expected to reduce government-wide annual costs attributable to physical data centers by a least 25 percent by the end of fiscal year 2018, resulting in savings of at least $2.7 billion.

About Half of the Agencies Planned to Meet OMB’s Targets for Data Center Closures

The 24 agencies reported mixed results regarding their data center closure progress and plans, when compared with OMB’s goal for each agency to close at least 25 percent of their tiered data centers and at least 60 percent of their non-tiered centers. Specifically, as of August 2018, 13 agencies reported that they had already met the goal of closing 25 percent of their tiered data centers, another 3 agencies reported that they planned to meet the goal by the end of fiscal year 2018, and 6 agencies reported that they did not plan to meet the goal.

Further, as of August 2018, 11 agencies reported that they had already met the goal for closing 60 percent of their non-tiered centers, 3 agencies reported that they planned to meet the goal by the end of fiscal year 2018, and 9 agencies reported that they did not plan to meet the goal by the end of fiscal year 2018. Table 1 displays a breakdown of the number of reported tiered and non-tiered data centers and completed and planned closures by agency, as of August 2018.

As shown in the figure below, the 24 agencies reported a total of 6,250 data center closures as of August 2018, which represented about half of the total reported number of federal data centers. In addition, the agencies planned 1,009 closures by the end of fiscal year 2018, with an additional 191 closures planned through fiscal year 2023 for a total of 1,200 more closures. This would further reduce the number of open data centers to about 39 percent of the number reported in the agencies’ inventories. Figure 3 provides a summary breakdown of agencies’ data center inventories that were closed, planned for closure, or not planned for closure, as of August 2018.

As noted, while about half of the agencies had met, or had planned to meet, their OMB targets as of August 2018, the other half planned to miss one or both of them. Officials from the 11 agencies that did not plan to meet one or both of their closure goals provided various reasons for why they had not planned to do so. For example, several agencies indicated that they were seeking revised closure goals because they viewed their goals as unattainable. Specifically, officials from Interior’s Office of the CIO stated that a number of the department’s non-tiered data centers were either mission-critical or not cost effective to close. Thus, the officials said Interior was working with OMB to establish a revised closure goal.

Similarly, Transportation’s Director for IT Compliance stated that the department was working with OMB to establish a revised closure goal. The department reported having 186 tiered data centers in Federal Aviation Administration control towers that it believes should be excluded from its count of data centers when OMB sets the department’s goal for closures. In addition, officials in Defense’s Office of the CIO stated that the OMB closure targets for the department were based on including special purpose processing nodes that are mission critical and, therefore, are not subject to being closed. The officials noted that the department intends to continue operating its enterprise data centers, close its smaller data centers, and work with OMB to remove the special purpose processing nodes from DCOI consideration.

When OMB launched DCOI in 2016, agencies originally had until the end of fiscal year 2018 to meet OMB’s stated time frame for closing their data centers. However, the extension of FITARA’s data center consolidation and optimization provisions through fiscal year 2020, and OMB’s planned revisions to DCOI goals, provide the 11 agencies that had not planned to meet one or both of OMB’s closure targets with additional time to meet their goals. Until these agencies take action to close enough data centers to meet OMB’s targets, they may not realize the efficiencies and cost savings that were expected from DCOI.

Almost Two-thirds of Agencies Planned to Meet OMB-Assigned Savings Targets

Since 2013, federal agencies have been required to report on data center cost savings, with guidance from OMB regarding how agencies were to report cost savings and avoidances. Specifically, the guidance required agencies to report both data center consolidation cost savings and avoidances, among other areas, as part of a quarterly reporting process. FITARA also called for each agency to submit a multi-year strategy for achieving the consolidation and optimization of data centers that included year-by-year quarterly calculations of investment and cost savings through fiscal year 2018, which has now been extended to 2020.

In addition, in August 2016, OMB M-16-19 provided guidance on how agencies should implement the requirements of FITARA. Specifically, agencies were to develop a strategic plan that included information on historical cost savings and avoidances due to data center consolidation and optimization through fiscal year 2015. This guidance stated that agency strategic plans were also to include year-by-year calculations of target and actual agency-wide spending and cost savings on data centers from fiscal years 2016 through 2018. Further, the guidance established a DCOI combined cost savings goal of $2.7 billion for all federal agencies to achieve from fiscal years 2016 through 2018. This overall goal was then broken down into agency-specific targets on the IT Dashboard.

In August 2018, 22 agencies reported through the quarterly reporting process that they had achieved $1.94 billion in cost savings for fiscal years 2016 through 2018, while 2 agencies reported that they had not achieved any savings. Further, 21 agencies identified an additional $0.42 billion planned through fiscal year 2018, for a total of $2.36 billion in planned savings from fiscal years 2016 through 2018. Nevertheless, this total is about $0.37 billion less than OMB’s goal of $2.7 billion for overall DCOI savings. Figure 4 compares the total achieved savings as reported by the 24 agencies for fiscal years 2016 through 2018 and the agencies’ additional planned savings through 2018 to OMB’s DCOI savings goal for fiscal years 2016 through 2018.

The 24 participating DCOI agencies had achieved $1.94 billion in savings as of August 2018. In addition, agencies identified an additional $0.43 billion, for a difference of $0.37 billion between planned and achieved savings from fiscal years 2016 through 2018. Table 2 provides specific data related to each agency’s total planned savings, total achieved savings, and additional planned savings through 2018.

As shown in table 2, 13 agencies reported that they had met or planned to meet or exceed their OMB targets, and 3 agencies that did not have an OMB target also identified achieved savings. In contrast, 5 agencies reported that they did not plan to meet their targets. Three agencies did not have a savings target and did not report any achieved savings.

Agencies provided various reasons for why they did not plan to meet their savings targets. For example, the Department of Veterans Affairs (VA) reported that the implementation of its DCOI-related projects in fiscal years 2016 through 2018 was dependent on funding approval and might not result in cost savings and avoidances until later in the projects’ life cycles (i.e., fiscal year 2019 or later). In another example, GSA stated that the OMB target may be difficult for the agency to reach due, in part, to the methods OMB used to set the target for fiscal years 2016 through 2018. According to GSA, OMB used the data that GSA had reported on the IT Dashboard regarding the agency’s expenditure for data center infrastructure and reduced that amount by 25 percent.

Agencies have now been working toward OMB’s DCOI savings goals since fiscal year 2016; however, almost half of the agencies are still not planning to meet OMB’s targets. Until agencies plan to meet and achieve OMB’s data center-related savings targets, they will likely not realize the expected financial benefits from DCOI.

Most Agencies Continued to Report Limited Progress Toward Meeting Optimization Metrics Targets

FITARA required OMB to establish metrics to measure the optimization of data centers, including server efficiency, and to ensure that agencies’ progress toward meeting the metrics is made public. Pursuant to FITARA, OMB established a set of five data center optimization metrics intended to measure agencies’ progress in the areas of server utilization and automated monitoring, energy metering, power usage effectiveness, facility utilization, and virtualization. According to OMB, while the server utilization and automated monitoring metric applied to agency-owned tiered and non-tiered data centers, the four remaining metrics applied only to agency-owned tiered centers.

OMB’s memorandum also established a target value for each of the five metrics, which agencies were expected to achieve by the end of fiscal year 2018. OMB measures agencies’ progress against the optimization targets using the agencies’ quarterly data center inventory submission and publicly reports this information on its IT Dashboard. Table 3 provides a description of the five data center optimization metrics and target values.

As of August 2018, most (22 of the 24) DCOI agencies continued to report limited progress in meeting OMB’s fiscal year 2018 data center optimization targets identified on the IT Dashboard. The remaining 2 agencies—Education and HUD—reported that they did not have any agency-owned data centers in their inventory and, therefore, did not have a basis to measure and report optimization progress.

With regard to the data center optimization targets, agencies reported the greatest progress against two metrics: power usage effectiveness and virtualization metrics. Specifically, 8 agencies reported that they had met OMB’s target for power usage effectiveness and 6 agencies reported that they had met the target for virtualization. However, for the energy metering, facility utilization, and server utilization and automated monitoring metrics, no more than 3 agencies reported meeting each. Figure 5 summarizes the 24 agencies’ progress in meeting each optimization target, as of August 2018.

As of August 2018, NSF, SSA, and EPA reported the most progress against OMB’s metrics among the 22 agencies with a basis to report— each met 3 targets. Nine agencies reported that they had met only one target, and 10 agencies reported they had not met any of the targets.

Further, OMB began requiring the implementation of automated monitoring tools in August 2016; however, as of August 2018, of the 22 agencies with a basis to report, 5 reported that they had either not implemented the tools at any data centers, or had experienced shortcomings in their implementation. For example, the Department of State (State) reported that it had limited centralized monitoring capability and is installing automated monitoring tools in several phases.

Thus, these 5 agencies were not able to report any progress against either or both of the server utilization or power usage effectiveness metrics because their data centers lacked the required monitoring tools to measure progress in these areas. The remaining 17 agencies reported that they had implemented the tools in at least one data center. Table 4 depicts the performance of the agencies in meeting OMB targets for data center optimization, as of August 2018.

As of August 2018, multiple agencies had made changes to their data center inventory and operational environment, such as closing all agency- owned tiered data centers or implementing automated monitoring tools. These changes impacted which metrics were applicable or an agency’s ability to report on the status of its optimization metrics. For example, GSA reported that it no longer had any agency-owned tiered data centers, and, therefore, did not have a basis to report on four of the five optimization metrics. Additionally, NSF, which previously had only owned one non-tiered data center, migrated from the non-tiered center to a tiered data center as part of its headquarters relocation. Accordingly, NSF began reporting on the metrics applicable to its tiered facility.

Further, the Nuclear Regulatory Commission (NRC) did not report on power usage effectiveness due to delays in awarding a new contract that was to include monitoring tools that would impact the ability to report on this metric. Agency officials stated that NRC plans to have the monitoring tools in place during fiscal year 2019. Overall, these changes since last year’s report have resulted in no significant changes to the progression of these agencies on their optimization metrics.

In addition, agencies’ limited progress against OMB’s optimization targets was due, in part, to not fully addressing our prior recommendations in this area. As previously mentioned, in March 2016, we reported on weaknesses in agencies’ data center optimization efforts, including that 22 agencies did not meet OMB’s fiscal year 2015 optimization targets. We noted that this was partially due to the agencies having challenges in optimizing their data centers, including in their decentralized organizational structures that made consolidation and optimization difficult, and in competing priorities for resources. In addition, consolidating certain data centers was problematic because the volume or type of information involved required the data center to be close in proximity to the users. Accordingly, we recommended that the agencies take action to improve optimization progress, to include addressing any of the identified challenges. Most agencies agreed with our recommendations or had no comments. However, as of December 2018, only 4 of the 22 agencies had fully addressed them.

The continuing shortcomings in data center optimization can also be attributed, in part, to agencies viewing OMB’s optimization metric targets as unrealistic. For example, Transportation stated in its DCOI strategic plan that it could not meet multiple optimization metrics due to funds not being available and competing priorities. In addition, Treasury indicated in its DCOI strategic plan that it struggles to report on automated monitoring because many of its data centers do not have the ability to centrally aggregate and report on central processing unit data. Further, DHS officials noted that it has 7 smaller tiered data centers where it has determined that it is not cost effective to equip those centers with the tools needed to report on metrics such as power usage effectiveness. Given these types of challenges, the targets for each optimization metric may not be realistic for every agency. Unless agencies take action to meet the applicable OMB optimization metrics, their data centers may not operate efficiently enough to provide expected cost savings.

Only Two Agencies Planned to Meet OMB’s Fiscal Year 2018 Optimization Targets

In addition to reporting current optimization progress on the IT Dashboard, OMB required agencies to include in their DCOI strategic plans planned performance levels for fiscal year 2018 for each optimization metric.

However, according to the 24 agencies’ DCOI strategic plan information as of August 2018, only 2—Commerce and the U. S. Agency for International Development (USAID)—reported plans to fully meet their applicable targets by the end of fiscal year 2018. Of the remaining agencies, 14 reported plans to meet some, but not all, of the targets; 6 reported that they did not plan to meet any targets, and—as already discussed—Education and HUD did not have a basis to report planned optimization milestones because they did not report having any agency- owned data centers. Figure 6 summarizes agencies’ progress, as of August 2018, in meeting OMB’s optimization targets and planned progress to be achieved by September 2018.

At the time of our review, only two agencies planned to meet all of their applicable targets, and it was doubtful that the agencies would be able to achieve OMB’s collective optimization target of at least $2.7 billion in cost savings by the end of fiscal year 2018. Until the remaining agencies take the steps necessary to meet their optimization targets, it is unlikely that these agencies will achieve the expected benefits of optimization and the resulting cost savings.

Selected Agencies Highlighted Successful DCOI Practices

As we noted previously in this report, many agencies have reported challenges that have hindered their efforts to meet OMB’s DCOI targets. However, a number of agencies have also reported success in meeting OMB’s targets ahead of DCOI’s end of fiscal year 2018 deadline. As noted in our methodology section, six agencies that were among the best performers in achieving data center closures, cost savings, and optimization performance reported a number of key practices that had contributed to their success. These practices were: obtaining executive leadership support for consolidation and using experiences and lessons learned to refine consolidation increasing the use of cloud and shared services to consolidate or optimize data center operations; emphasizing closing data centers to meet OMB targets and achieve increasing the use of virtualization to optimize data centers; and employing an organization-wide communications plan to facilitate adoption of consolidation and optimization activities.

Obtaining Executive Leadership Support for Consolidation and Optimization Activities

Five of the six agencies (Agriculture, Commerce, Justice, EPA, and GSA) reported that their success in consolidation and optimization activities was due to obtaining support from executive leadership for the agency’s consolidation efforts. Each agency obtained sponsorship and support from its executive leadership (e.g., Deputy Secretary or agency CIO), such as through a memorandum or policy that directed all agency offices to participate in, or comply with, the consolidation effort. For example,

The Deputy Secretary for Agriculture issued a memorandum in 2017 that, among other things, declared the department’s intent to consolidate from 39 data centers down to 2 by the end of 2019. According to officials in the Office of the CIO, this memorandum from the Secretary’s office focused all data center owners on the same project task of reducing the data center inventory.

The Commerce CIO and the department’s CIO Council provided overall governance through organizational policies, processes, and procedures for the department’s data center consolidation effort. Leveraging this departmental guidance, each component of Commerce developed its own consolidation plan that identified specific approaches and activities. Using these plans, the department and its components focused on reducing spending on redundant software, infrastructure, and data center operations.

The Deputy Attorney General issued a memorandum in 2014 to the heads and CIOs of all components. This memorandum formally established Justice’s Data Center Transformation Initiative, established the Department Program Review Board to provide oversight for the initiative, and also directed the consolidation of all data centers into 3 enterprise facilities. In addition, Justice’s CIO issued a memorandum to component CIOs that provided additional details on how to execute planned activities and established further governance associated with the initiative. These memoranda provided clear leadership buy-in and support for the department’s data center consolidation and optimization activities that could be used to resolve any challenges or issues at the departmental level.

EPA attributed much of its DCOI success to a top-down approach from its CIO office, saying that such support was critical to achieve data center closures. For example, EPA leadership decided to adopt and enforce geographical consolidation of data centers within major areas to minimize costs of consolidation while still meeting closure objectives. In doing so, the agency leadership provided clear direction and support for the agency’s consolidation effort by adopting the strategy to consolidate data centers within specific geographic regions.

GSA reported that it obtained leadership commitment that made its data center consolidation and optimization activities a priority. The agency noted that having strong CIO and executive leadership was important for sponsoring technology modernization. As a result of the buy-in, the agency reported that it had minimized resistance to change and improved acceptance of its consolidation and optimization activities.

Using Experiences and Lessons Learned to Refine Consolidation Planning

Four agencies (Agriculture, Commerce, Justice, and SSA) reported that their success with consolidation and optimization activities was due to the use of a refined consolidation plan or process. Each of these agencies developed an initial consolidation plan or process for closing data centers, and then refined their procedures based on their experiences and lessons learned as data centers were closed. For example,

Agriculture developed a set of streamlined processes to facilitate DCOI closures that were based on the experiences gained from successful data center closures under FDCCI. The set of processes consisted of 5 steps:

The planning step included the discovery and documentation of all data center assets, including applications and IT hardware, in a given data center. In addition, this step involved identifying the necessary resources to move the applications and associated data to a target data center.

The preparation step included identification of the target data center and development of a project schedule.

The data migration step included moving both applications and data to the target data center or cloud-services, as planned.

The testing step included ensuring the applications and data that were moved were integrated into the target data center, and functional testing to ensure that the applications worked and data was accessible.

The application cutover step included putting the migrated applications and data into operation and closing the original data center.

Using and refining this set of processes allowed the department to become more efficient in closing its data centers. After closing 46 data centers in fiscal years 2011 through 2014, the department closed 2,185 data centers over the next 2 years. In total, Agriculture reported that it had closed 2,253 data centers as of August 2018.

Commerce established departmental guidance and then each departmental component leveraged that guidance to develop its own consolidation plan. The plans identified specific approaches and activities intended to achieve the stated goals and milestones. According to Commerce, the department and its components leveraged their IT planning processes and established IT governance to, among other things, reduce spending on redundant commodity software, infrastructure, and operations.

Justice’s Office of the CIO developed a master plan for the department’s data center consolidation effort in June 2015. The plan included a planning framework, transformation approach, and a master schedule for data center moves and closures. It also included process steps similar to those used by Agriculture. Further, Justice’s plan noted that the department would use its initial closure efforts to gain experience and to refine its plans. Justice reported that it used the plan’s schedule and semi-monthly progress reports to ensure that consolidation activities stayed on schedule, or the department could make adjustments as needed. As a result, the department closed 84 of its 110 data centers and achieved more than $128 million in cost savings and avoidances as of August 2018.

SSA used a project management framework process and controls that it believed efficiently addressed requirements, critical path, and risk management. In addition, SSA reported that it used an incremental development approach to its data center optimization plans, with each project expected to accomplish specific tasks that would lead to another project. Accordingly, SSA noted that the agency used a multi-year plan with many initiatives focused on specific goals. Using this approach, the agency successfully moved SSA’s operations and infrastructure from an older facility to the newly-built National Support Center. The agency reports that this facility is state-of-the-art and provides similar capabilities and efficiencies to major cloud service providers.

Increasing the Use of Cloud and Shared Services to Consolidate or Optimize Data Center Operations

Three agencies (Commerce, GSA, and SSA) also attributed their success in consolidation and optimization activities to increasing their agency’s use of cloud and shared services. In doing so, each agency emphasized the move of data center assets and systems to cloud services to optimize their data centers and reduce costs. For example,

Commerce identified moving to cloud services and utilizing shared services as being most effective in closing data centers. As an example, the department cited the National Oceanic and Atmospheric Administration’s (NOAA) “cloud-first” policy that emphasized using cloud services rather than an agency-owned physical data center whenever feasible. The agency attributed its ability to handle increased traffic as an operational benefit of its increased use of cloud services. For example, NOAA did not have the capacity in its agency- owned facilities to meet the computing demands and requirements of a sudden increase in web traffic on the websites for NOAA and the National Hurricane Center, such as during Hurricanes Irma and Harvey in 2017. Commerce stated that using cloud services allowed NOAA to handle 4.7 billion page hits during Hurricane Harvey over a 6-day span, ensuring the websites were not adversely impacted by the increase in traffic.

GSA reported that it focused on moving services from agency-owned tiered and non-tiered data centers to cloud services or to shared centers. As a result, GSA had closed 118 data centers as of August 2018, including all of the agency’s tiered centers.

SSA developed an agency cloud initiative that encourages the adoption of cloud technologies as part of the agency’s infrastructure modernization. The agency reported that it is employing a hybrid cloud strategy that is comprised of both private cloud and public cloud services for the agency’s back office applications. By doing so, the agency will consolidate and standardize SSA’s IT infrastructure systems and software to simplify management of those resources and reduce costs.

Emphasizing the Closure of Data Centers to Meet OMB Targets and Achieve Cost Savings

Three agencies (Agriculture, Justice, and EPA) reported that their success in consolidation and optimization activities was due to focusing on the closure of data centers. In doing so, they emphasized the importance of closing data centers to reduce costs and achieve cost savings and avoidances. For example,

Agriculture determined that the costs to improve DCOI performance metrics in its agency-owned data centers were prohibitive. Accordingly, the department decided that the only viable alternative was to close data centers to remove underperforming centers and improve optimization metrics performance and reduce costs. As a result, Agriculture reported that it had closed 2,253 data centers through August 2018. In addition, the department reported that it had improved its security posture, reduced its real estate footprint, and achieved realized cost savings and avoidance of $51.8 million from fiscal year 2012 through 2018.

Justice reported that it took a practical approach to selecting the data centers that would remain as its enterprise facilities, considering factors such as the number of physical servers that could be eliminated, the efficiency of the remaining hardware, and potential labor savings. The department reported that it focused on retaining more efficient data centers (e.g., those with more efficient use of electricity or virtualization), rather than simply keeping its biggest existing data centers. As a result, Justice has closed 84 of its 110 data centers and achieved more than $128 million in cost savings and avoidances as of August 2018.

EPA identified geographical consolidation as its best approach to meeting DCOI goals. Specifically, in its data center consolidation plan, the agency stated that, for geographic areas where it had multiple data centers, a single facility was identified into which data center IT assets would be consolidated. Using this approach, EPA had closed 43 of its 83 data centers as of August 2018.

Increasing the Use of Virtualization to Optimize Data Centers

Three agencies (Commerce, EPA, and SSA) reported that their success in consolidation and optimization activities also was due to focusing on the increased use of virtualization to run more software on the same or a reduced amount of servers. In doing so, the agencies expected to reduce costs by avoiding the purchase of additional servers to meet computing demands or eliminating unnecessary hardware and floor space in their data centers. For example,

Commerce focused on moving systems from physical hardware to virtual servers, as part of its component offices’ plans to update technology and in cases where the systems did not require a specific type of server. Using this approach, the department reported that it had reduced the number of physical servers in its data centers, and was working to improve server utilization. The department also cited the ability to automatically increase or decrease computing capability through virtualization, such as when NOAA handled the increased traffic to its hurricane-related web pages during Hurricanes Irma and Harvey in 2017.

EPA used the agency’s data center consolidation plan to implement an agency-wide “physical-to-virtual” policy that required offices to convert existing physical servers to virtual servers wherever possible. The agency also defined server and software standards for virtualized platforms.

SSA reported that the agency’s goal, using its “Virtual 1st” policy, was to have failover capability within the data center, disaster recovery capability for both data centers, and balanced load capacity between data centers. The agency reported that it has continued to virtualize not only servers but storage and network applications, as well. For example, SSA stated that it has taken steps to virtualize as much storage as possible and used similar techniques to reduce the physical hardware footprint on the data center floor, as well as power, cooling, and network bandwidth requirements.

Employing an Organization-wide Communications Plan to Facilitate Adoption of Consolidation and Optimization Activities

Two agencies (Justice and GSA) reported that their success in consolidation and optimization activities was due to employing an organization-wide communications plan. In doing so, the agencies adopted a structured method for communicating with agency offices to improve acceptance and adoption of consolidation and optimization activities. This also facilitated conflict resolution. For example, Justice reported that it prioritized communications related to its Data Center Transformation Initiative and established an all-encompassing approach to initiative-related communications. To help communicate all related directives, strategies, plans, statuses, and accomplishments, the department used a variety of methods that included: regular meetings to share information, a dedicated email box to provide easy communication for answers or information, without the need to know specific individuals, an intranet web page that provided general information, instructions, templates, decisions, status information, and accomplishments related to the initiative; and email broadcasts on an as-needed basis.

GSA reported that it communicated and collaborated frequently with business stakeholders to identify the best time frames to move systems, stagger transfers to minimize impact, and determine which systems could be virtualized. The agency indicated that these important factors required continuous communication between system owners, system administrators, and business leadership. As a result, the agency experienced minimal staff resistance to change and a commitment to reach a consensus on moving forward with the agency’s consolidation efforts.

The aforementioned practices included elements of sound management techniques, such as gathering leadership support for a project and developing a communications plan to foster adoption of organizational changes. The practices also included activities that aligned with the core tenets of DCOI to consolidate inefficient infrastructure, optimize existing facilities, and achieve cost savings. Further, these practices each proved effective for multiple agencies and, while they were not the only practices that could be effective, they represent concepts that could provide the foundation for an effective data center consolidation and optimization program.

Conclusions

Federal data center consolidation efforts have been underway since 2010 and OMB’s fiscal year 2018 targets provided clear and transparent goals that helped define the tangible benefits that DCOI was expected to provide. However, most agencies continue to report mixed progress against those targets. Although agencies have taken action to close about half of the data centers in their combined inventories, 11 agencies did not plan to meet all of their closure targets.

Further, the data center closures were expected to drive cost savings and avoidances and, to the agencies’ credit, the closures have led to more than $2.37 billion in planned and achieved cost savings and avoidances from fiscal years 2016 through 2018. However, five agencies did not plan to meet their cost savings targets. Until agencies consolidate the data centers required to meet their targets, as well as identify and report the associated cost savings, they will be challenged to realize expected efficiencies and the full benefits of DCOI will not be fully realized.

Similarly, although OMB first established optimization metrics in May 2014, agencies continue to report only limited progress against the current performance targets. While two agencies do not have a basis to report any progress as they do not own any data centers, only two agencies reported that they planned to achieve all of DCOI’s fiscal year 2018 optimization targets. Ensuring the optimized performance of data centers is a key component to meeting OMB’s DCOI-wide savings goal and the 20 agencies that did not have plans to meet their targets call into question whether DCOI will realize its full potential savings.

Although many agencies have struggled to meet their individual DCOI targets, other agencies have successfully met OMB’s goals for data center closures, savings, and optimization. Six such agencies that we identified reported on the importance of gathering leadership support, effective communication, and alignment with the core tenets of DCOI. Key practices such as these can play an important role in helping agencies better meet the overall goals and mission of DCOI.

Recommendations

We are making a total of 36 recommendations to 22 of the 24 agencies in our review. Specifically: The Secretary of Agriculture should take action to meet the data center optimization metric targets established by OMB under DCOI. (Recommendation 1)

The Secretary of Commerce should take action to meet the data center closure targets established under DCOI by OMB. (Recommendation 2)

The Secretary of Defense should take action to meet the data center closure targets established under DCOI by OMB. (Recommendation 3)

The Secretary of Defense should identify additional savings opportunities to achieve the targets for data center-related cost savings established under DCOI by OMB. (Recommendation 4)

The Secretary of Defense should take action to meet the data center optimization metric targets established under DCOI by OMB. (Recommendation 5)

The Secretary of Energy should take action to meet the data center closure targets established under DCOI by OMB. (Recommendation 6)

The Secretary of Energy should take action to meet the data center optimization metric targets established under DCOI by OMB. (Recommendation 7)

The Secretary of the Department of Health and Human Services (HHS) should take action to meet the data center closure targets established under DCOI by OMB. (Recommendation 8)

The Secretary of HHS should take action to meet the data center optimization metric targets established under DCOI by OMB. (Recommendation 9)

The Secretary of DHS should take action to meet the data center closure targets established under DCOI by OMB. (Recommendation 10)

The Secretary of DHS should take action to meet the data center optimization metric targets established under DCOI by OMB. (Recommendation 11)

The Secretary of Interior should take action to meet the data center closure targets established under DCOI by OMB. (Recommendation 12)

The Secretary of Interior should take action to meet the data center- related cost savings established under DCOI by OMB. (Recommendation 13)

The Secretary of Interior should take action to meet the data center optimization metric targets established under DCOI by OMB. (Recommendation 14)

The Attorney General should take action to meet the data center optimization metric targets established for Justice under DCOI by OMB. (Recommendation 15)

The Secretary of the Department of Labor (Labor) should take action to meet the data center optimization metric targets established under DCOI by OMB. (Recommendation 16)

The Secretary of State should take action to meet the data center closure targets established under DCOI by OMB. (Recommendation 17)

The Secretary of State should take action to meet the data center optimization metric targets established under DCOI by OMB. (Recommendation 18)

The Secretary of Transportation should take action to meet the data center closure targets established under DCOI by OMB. (Recommendation 19)

The Secretary of Transportation should take action to meet the data center optimization metric targets established under DCOI by OMB. (Recommendation 20)

The Secretary of Treasury should take action to meet the data center optimization metric targets established under DCOI by OMB. (Recommendation 21)

The Secretary of VA should take action to meet the data center closure targets established under DCOI by OMB. (Recommendation 22)

The Secretary of VA should take action to meet the data center-related cost savings established under DCOI by OMB. (Recommendation 23)

The Secretary of VA should take action to meet the data center optimization metric targets established under DCOI by OMB. (Recommendation 24)

The Administrator of EPA should take action to meet the data center closure targets established under DCOI by OMB. (Recommendation 25)

The Administrator of EPA should take action to meet the data center optimization metric targets established under DCOI by OMB. (Recommendation 26)

The Administrator of GSA should take action to meet the data center optimization metric targets established under DCOI by OMB. (Recommendation 27)

The Administrator of the National Aeronautics and Space Administration (NASA) should take action to meet the data center optimization metric targets established under DCOI by OMB. (Recommendation 28)

The Director of NSF should take action to meet the data center optimization metric targets established under DCOI by OMB. (Recommendation 29)

The Chairman of NRC should take action to meet the data center optimization metric targets established under DCOI by OMB. (Recommendation 30)

The Director of OPM should take action to meet the data center-related cost savings established under DCOI by OMB. (Recommendation 31)

The Director of OPM should take action to meet the data center optimization metric targets established under DCOI by OMB. (Recommendation 32)

The Administrator of the Small Business Administration (SBA) should take action to meet the data center optimization metric targets established under DCOI by OMB. (Recommendation 33)

The Commissioner of SSA should take action to meet the data center- related cost savings established under DCOI by OMB. (Recommendation 34)

The Commissioner of SSA should take action to meet the data center optimization metric targets established under DCOI by OMB. (Recommendation 35)

The Administrator of USAID should take action to meet the data center closure targets established under DCOI by OMB. (Recommendation 36)

Agency Comments and Our Evaluation

We requested comments on a draft of this report from OMB and the 24 agencies that we reviewed. Of the 22 agencies to which we made recommendations, 11 agencies agreed with our recommendations; three agencies agreed with some portion, but not all of the recommendations; one agency disagreed with our recommendations; and seven agencies did not state whether they agreed or disagreed with the recommendations. In addition, OMB and two agencies to which we did not make recommendations stated that they had no comments. Further, multiple agencies provided technical comments, which we have incorporated, as appropriate.

The following 11 agencies agreed with our recommendations: In written comments from Commerce, State, NASA, SBA, and SSA, the agencies stated that they agreed with the recommendations and indicated their intent to address them. State also provided technical comments, which we have incorporated, as appropriate. The agencies’ comments are reprinted in appendices II through VI.

In written comments, Energy agreed with our recommendations to meet its data center closure and optimization metric targets, and described actions that the department planned to take in order to address the recommendations. Energy initially estimated that it would complete these actions by March 1, 2019; however, the department subsequently revised its estimated completion date to April 15, 2019. Energy also provided technical comments, which we have incorporated, as appropriate. Energy’s comments are reprinted in appendix VII.

In written comments, VA agreed with our recommendations to meet its data center closure, cost savings, and optimization metric targets. In addition, the department requested that we close our recommendation related to data center closures on the basis of its planned actions to implement a new inventory data collection tool and methodology to improve how the department collects data center inventory information, and a positive trend in its data center closures.

The department estimated that its planned actions would be completed in March 2019 and reported that, as of November 2018, it had closed 78 data centers in fiscal year 2018, as compared with 24 in fiscal year 2017.

However, as noted earlier in this report, we found that VA did not plan to meet the closure goal for either tiered or non-tiered data centers, which was the basis for our recommendation. While we acknowledge and encourage VA’s reported closure progress, the department still has not met its DCOI closure goals, as we recommended. Further, VA did not provide an update on the status of its planned actions in time for us to address them in this report. As such, we maintain that this recommendation is still appropriate.

In addition, VA referred to OMB’s proposed changes to DCOI guidance when describing actions that it planned to take to meet the department’s cost savings and optimization metrics targets. However, OMB staff told us that the August 2016 DCOI guidance will remain in effect until the revised DCOI guidance is formally issued. Once OMB’s new DCOI guidance is finalized, we plan to assess agency progress against any revised targets, and we will continue to monitor the department’s efforts to address our recommendation. VA’s comments are reprinted in appendix VIII.

We received emails from officials of Agriculture, Justice, Transportation, and OPM which stated that these agencies agreed with the recommendations we directed to them. In addition, three agencies agreed with some portion, but not all of our recommendations directed to them: In written comments, Defense stated that it agreed with our recommendation to meet its data center closure targets. However, the department partially agreed with our two other recommendations: to identify additional data center-related savings opportunities and to meet OMB’s data center optimization metric targets.

In partially agreeing with our recommendation on data center savings, Defense asserted that it had already identified significant cost savings through activities such as the identification of system migration candidates and the use of cloud services, among others. The department further stated that, while it would continue to optimize its data centers, the need for IT would continue to grow, and this growth might ultimately lead to an increase in total data center costs, despite overall per unit cost reductions.

However, the department’s planned savings of $205.46 million represented only 11 percent of its $1.8 billion savings goal by the end of fiscal year 2018 and, as such, this limited progress by the department formed the basis for our recommendation. As discussed in our report, OMB plans to revise DCOI guidance and work with agencies to set agency-specific targets. According to OMB staff, until the guidance is revised, the current guidance and its targets are still applicable. For these reasons, we maintain that our recommendation is still appropriate.

Further, in partially agreeing with our recommendation to meet optimization metric targets, Defense stated that the department will continue to drive towards the achievement of data center optimization targets. It added, however, that it would not invest resources to improve the efficiency of data centers planned for closure and that, as a result, the composite view of Defense’s data center efficiency would fall short of meeting OMB’s targets.

Our review found that Defense did not plan on meeting any of OMB’s five data center optimization metric targets by the end of fiscal year 2018. This finding was the basis for our recommendation. We acknowledge Defense’s position that investing resources into optimizing data centers that are already planned for closure would not be the best use of taxpayer dollars. We also noted in our report that OMB had proposed revising its optimization metrics, and that any such changes had not yet been finalized. Our recommendation is not intended to imply that an agency should meet a particular version of OMB targets but, rather, that the agency should meet any targets that are established by OMB. This would include any future changes to DCOI targets. Accordingly, we maintain that our recommendation is still appropriate and will continue to monitor the department’s efforts to address our recommendation. Defense’s comments are reprinted in appendix IX.

In written comments, DHS stated that it agreed with our recommendation to meet its data center closure targets and disagreed with our recommendation to meet its data center optimization metric targets. Specifically, the department noted that it had met its tiered data center closure targets, and was reviewing the status of its remaining open non-tiered data centers. The department added that it expected to complete this activity by March 31, 2019. However, the department did not provide an update on its efforts in time to be included in this report.

While we encourage DHS’s continued efforts to close its remaining non-tiered data centers, we note that the department’s letter cites an inventory of 18 open non-tiered facilities, which differs significantly from the 202 non-tiered centers counted in our draft report, and which DHS officials confirmed in November 2018. According to the department, this discrepancy is because OMB issued revised inventory reporting requirements in November 2018, and these revised requirements exempted certain types of facilities from DCOI reporting and resulted in the lower number.

These changes in reporting requirements are similar to the proposed, but not yet finalized, revisions to the DCOI policy that are discussed earlier in this report. However, OMB staff told us that the August 2016 DCOI guidance will remain in effect until the revised DCOI guidance is formally issued. Once OMB’s new DCOI guidance is finalized, we plan to assess agency progress against any revised targets, and we will continue to monitor the department’s efforts to address our recommendation.

Further, in disagreeing with our recommendation on meeting optimization metrics, the department stated that, while the recommendation was applicable under the original DCOI guidance that OMB issued in August 2016, OMB’s proposed changes to DCOI guidance would exempt most, if not all, DHS agency-owned data centers from the optimization metrics. Consequently, the department requested that our recommendation be closed.

In our review, we found that the department did not plan on meeting any of OMB’s five data center optimization metric targets established under DCOI. This finding was the basis for our recommendation on meeting optimization metrics. Also, while OMB has proposed changes to its metrics, as we noted previously, it has not provided a date for when any such proposed changes will be finalized and implemented; and, according to OMB staff, until the changes to DCOI guidance are finalized, the current guidance is still applicable. Further, our recommendations do not specify that an agency should meet any particular version of OMB targets, but rather, that an agency should meet the targets established by OMB. This would include any future changes to DCOI targets. Accordingly, we maintain that our recommendation is still appropriate. DHS also provided technical comments, which we have incorporated, as appropriate. DHS’s comments are reprinted in appendix X.

In written comments, Interior stated that it partially agreed with our recommendation to meet its data center closure targets and disagreed with our two recommendations to meet its data center-related cost savings target and its data center optimization metric targets. For all three recommendations, the department stated that OMB had proposed changes to DCOI guidance that would result in new targets for closures, cost savings, and optimization metrics and that Interior planned to adopt the new policy and work through OMB to establish its new targets.

As noted in our report, Interior met its target for tiered data center closures, but did not plan to meet the closure goal for non-tiered data centers. Further, the department planned on achieving only $15.95 million of its $88.19 million savings target (18 percent) by the end of fiscal year 2018, and did not plan on meeting any of OMB’s five data center optimization metric targets. These three findings were the basis for our recommendations to the department.

We also noted that, as part of OMB’s proposed changes to DCOI guidance, it planned to work with agencies to set agency-specific targets for data center closures and planned to modify the metrics currently used by agencies to monitor the performance of their data centers. However, as previously mentioned, OMB has not provided a date for when these proposed changes will be finalized and implemented and, according to OMB staff, until the changes to DCOI guidance are finalized, the 2016 guidance is still applicable. Furthermore, our recommendations do not specify that an agency should meet any particular version of OMB targets, but should meet any targets that are established by OMB. This would include any future changes to DCOI targets. As such, we maintain that our recommendations are appropriate. Interior’s comments are reprinted in appendix XI.

One agency disagreed with all of our recommendations: In written comments, HHS disagreed with our two recommendations to meet its data center closure targets and data center optimization metric targets. In regard to both recommendations, the department disagreed with being held to what it termed “expired requirements” from DCOI guidance, pending the assignment of new targets being established by OMB.

As noted in our report, HHS met its target for tiered data center closures, but did not plan to meet the closure target for non-tiered data centers. We also found that HHS did not meet any of OMB’s five optimization metric targets and had planned to meet only one of the five by end of fiscal year 2018. These findings were the basis for the two recommendations that we made to the department.

We also noted that, as part of OMB’s proposed changes to DCOI guidance, OMB planned to work with agencies to set agency-specific targets for data center closures and planned to modify the metrics currently used by agencies to monitor the performance of their data centers. However, as previously mentioned, OMB did not provide a date for when these proposed changes will be finalized and implemented and, according to its staff, until the changes to DCOI guidance are finalized, the current guidance is still applicable. Further, our recommendations do not specify that an agency should meet any particular version of OMB targets, but rather, that the agency should meet the targets established by OMB. This would include any future changes to DCOI targets. Accordingly, we maintain that our recommendations are still appropriate. HHS’s comments are reprinted in appendix XII.

Further, seven agencies did not agree or disagree with the recommendations: In written comments, EPA did not state whether it agreed or disagreed with our recommendations to meet its data center closure and data center optimization metrics targets. However, the agency requested that we close our recommendations, citing its reported progress in closing 21 of 34 targeted data centers and OMB’s proposed changes in its draft DCOI guidance that could result in revised closure targets and optimization metrics.

As stated in our report, we found that EPA did not plan to meet its closure target for tiered or non-tiered data centers, nor did it plan to meet its data center optimization targets; these findings were the basis for our recommendations. We also noted that, as part of OMB’s proposed changes to DCOI, OMB planned to work with agencies to set agency-specific targets for data center closures and planned to modify the metrics currently used by agencies to monitor the performance of their data centers. However, OMB has not provided a date for when these proposed changes will be finalized and implemented and, according to OMB staff, until the changes to DCOI guidance are finalized, the current guidance is still applicable. Further, our recommendations do not specify that an agency should meet any particular version of OMB targets, but that it should meet the targets established by OMB. This would include any future changes to DCOI targets. Accordingly, we maintain that our recommendations are appropriate and should remain open. EPA also provided technical comments, which we have incorporated, as appropriate. The agency’s comments are reprinted in appendix XIII.

In written comments, GSA did not state whether it agreed or disagreed with our recommendation to meet the agency’s data center optimization metrics targets. Specifically, the agency stated that it had complied with revised inventory reporting requirements, which OMB provided to agencies in November 2018 and which eliminated non- tiered data centers from the requirement to meet optimization targets. As a result, the agency noted that it no longer had a basis to measure and report on the one metric our report cited as applicable to GSA (i.e., server utilization and automated monitoring) and asked that we withdraw the recommendation.

These changes in reporting requirements are similar to the proposed, but not yet finalized, revisions to the DCOI policy that are discussed earlier in the report. However, OMB staff told us that the August 2016 DCOI guidance is still in effect until the revised DCOI guidance is formally issued. Until OMB’s new DCOI guidance is finalized and agency progress against any revised targets can be evaluated, we maintain that our recommendation to meet the agency’s optimization metrics targets is appropriate, and we will continue to monitor the agency’s efforts to address it. GSA’s comments are reprinted in appendix XIV.

In written comments, NSF did not state whether it agreed or disagreed with our recommendation. The agency’s comments are reprinted in appendix XV.

In written comments, NRC agreed with the draft report, but did not state whether it agreed or disagreed with our recommendation. The agency’s comments are reprinted in appendix XVI.

In written comments USAID did not state whether it agreed or disagreed with the draft report’s recommendation but agreed with our finding that the agency no longer had any tiered data centers. However, USAID stated that it had met DCOI’s closure targets for the agency by closing its 4 non-tiered data centers, and requested that we close our recommendation to meet those targets.

While we encourage USAID’s continued efforts to close its remaining non-tiered data centers, we note that the agency’s letter cites an inventory of 4 non-tiered facilities, which differs significantly from the 83 non-tiered centers counted in our draft report, and which USAID officials confirmed in October 2018. As USAID communicated in subsequent emails, this discrepancy is because OMB issued revised inventory reporting requirements in November 2018 and these revised requirements exempted certain types of facilities from DCOI reporting, which resulted in the lower number.

These changes in reporting requirements are similar to the proposed, but not yet finalized, revisions to the DCOI policy that are discussed earlier in the report. However, OMB staff told us that the August 2016 DCOI guidance is still in effect until the revised DCOI guidance is formally issued. Until OMB’s new DCOI guidance is finalized and agency progress against any revised targets can be evaluated, we maintain that our data center closure recommendation is appropriate, and we will continue to monitor the agency’s efforts to address it. The agency’s comments are reprinted in appendix XVII.

In emails received from Labor’s GAO liaison in the department’s Office of the Assistant Secretary for Policy on January 8, 2019, and from an audit liaison in Treasury’s Office of the CIO on February 1, 2019, both departments did not state whether they agreed or disagreed with our respective recommendations.

Finally, in emails received from a Management and Program Analyst in Education’s Office of the Secretary/Executive Secretariat on January 8, 2019; an audit liaison in HUD’s Office of the CIO, Audit Compliance Branch on February 15, 2019; and a GAO liaison in OMB’s Office of General Counsel on February 25, 2019, these agencies stated that they had no comments on the draft report.

We are sending copies of this report to interested congressional committees, the Director of OMB, the secretaries and heads of the departments and agencies addressed in this report, and other interested parties. In addition, the report will be available at no charge on GAO’s website at http://www.gao.gov.

If you or your staffs have any questions about this report, please contact me at (202) 512-4456 or harriscc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix XVIII.

Appendix I: Objectives, Scope, and Methodology

Our objectives for this engagement were to (1) determine agencies’ progress in data center closures and achievement in related savings to date and describe plans for future savings, (2) evaluate the agencies’ progress against OMB’s data center optimization targets, and (3) identify effective agency practices for achieving data center closures, cost savings, and optimization.

To address the first objective, for data center closures, we obtained and analyzed August 2018 data center inventory documentation from the 24 departments and agencies (agencies) that participate in OMB’s Data Center Optimization Initiative (DCOI). To determine data center closures to date, we totaled their reported closures from fiscal year 2010 through August 2018 and to identify future closures, we totaled their reported planned closures through fiscal year 2018. We also compared agencies’ completed and planned closures to OMB’s fiscal year 2018 consolidation goals, as documented in its August 2016 memorandum (M-16-19).

To verify the quality, completeness, and reliability of each agency’s data center inventory, we compared information on completed and planned data center closures to similar information reported on OMB’s IT Dashboard—a public website that provides information on federal agencies’ major IT investments. We also checked for missing data and other errors, such as missing closure status information. In some cases identified, we followed-up with agency officials to obtain further information. We determined that the data were sufficiently complete and reliable to report on agencies’ consolidation progress and planned closures.

For cost savings and avoidance we obtained and analyzed documentation from the 24 DCOI agencies. This documentation is required by OMB’s March 2013 and August 2016 memorandums and included the agencies’ quarterly reports of cost savings and avoidances posted to their digital services websites and their DCOI strategic plans. To determine cost savings achieved, we totaled agencies’ reported savings and avoidances from the start of fiscal years 2012 through August 2018, as found in the August 2018 quarterly reports posted to the agencies’ digital services websites. To identify future planned savings, we totaled the agencies’ projected savings and avoidances from fiscal years 2016 through 2018, as reported in their DCOI strategic plans.

To assess the quality, completeness, and reliability of each agency’s data center consolidation cost savings information, we used the latest version of each agency’s quarterly cost savings report and DCOI strategic plan, as of August 2018. We also reviewed the quarterly reports and DCOI strategic plans for missing data and other errors, such as missing cost- savings information. In addition, we compared agencies cost savings and avoidances with data from our most recent data center consolidation report. As a result, we determined that the data were sufficiently complete and reliable to report on agencies data center consolidation cost-savings information.

For our second objective, we analyzed the August 2018 data center optimization progress information of the 24 DCOI agencies. This progress information was obtained from the IT Dashboard—an OMB public website that provides information on federal agencies’ major IT investments. To assess agencies’ planned optimization progress, we obtained the planned optimization performance from the 22 agencies’ DCOI strategic plans. We then compared the agencies’ current and planned optimization progress information to OMB’s fiscal year 2018 optimization targets, as documented in its August 2016 memorandum. Although OMB’s memorandum establishes a single optimization target value for the server utilization and automated monitoring metric, the IT Dashboard displays agencies’ progress for tiered and non-tiered data centers separately. To report consistently with OMB’s implementation memorandum, we combined the progress information for tiered and non-tiered data centers into a single assessment in this report.

In addition, to assess the reliability of the planned optimization milestones in the DCOI strategic plans, we reviewed agencies’ documentation to identify any missing or erroneous data. We also compared the planned data center optimization milestones contained in agencies’ documentation against current optimization progress information obtained from the IT Dashboard; we then discussed any discrepancies or potential errors that we identified with agency officials to determine the causes or request additional information. As a result of these efforts, we were able to determine whether each agency’s strategic plan information was sufficiently reliable for reporting on plans to meet or not meet OMB’s fiscal year 2018 optimization targets.

To assess the reliability of agencies’ optimization progress information on OMB’s IT Dashboard, we reviewed the information for errors or missing data, such as progress information that was not available for certain metrics. We also compared agencies’ optimization progress information across multiple reporting quarters to identify any inconsistencies in agencies’ reported progress. We discussed with staff from OMB’s Office of the Federal Chief Information Officer any discrepancies or potential errors identified to determine the causes.

To identify effective agency practices for achieving data center closures, cost savings, and optimization progress, we selected two of the highest performing departments or agencies for each of those three data center areas that we reported on in our May 2018 report. For the data center inventory closures area, we selected the Departments of Agriculture (Agriculture) and Justice (Justice) from among the five agencies that had, as of August 2017, reached or exceeded both their tiered and non-tiered data center closure targets for the end of fiscal year 2018. For the cost savings area, we identified two departments and two small agencies reporting the highest cost savings DCOI to date, as of August 2017. From those, we selected one department (Commerce) and one small agency (the General Services Administration) to provide balance relative to agency size. For effective practices related to optimization performance, we reviewed agencies’ reported optimization performance as of August 2017 and selected the two highest-performing agencies in this area (the Social Security Administration and the Environmental Protection Agency), since they were the only two agencies reporting that they met more than half of OMB’s optimization targets. Selecting these agencies was designed to provide anecdotal information that could assist agencies struggling with DCOI implementation. The examples they provided are not findings nor should they be taken to be representative of all the agencies participating in DCOI.

We asked each selected agency to identify practices that they found effective in implementing DCOI at their agency and in meeting OMB’s established targets in each of the areas, not just the area for which they were selected. We also solicited examples that demonstrated how those practices helped agency implementation or the benefits from implementing DCOI. Additionally, we considered information and examples that these agencies provided as part of our work to identify FITARA best practices. We analyzed the responses to determine the practices and reported those that were identified by at least two agencies.

We conducted this performance audit from April 2018 to April 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Department of Commerce

Appendix III: Comments from the Department of State

Appendix IV: Comments from the National Aeronautics and Space Administration

Appendix V: Comments from the Small Business Administration

Appendix VI: Comments from the Social Security Administration

Appendix VII: Comments from the Department of Energy

Appendix VIII: Comments from the Department of Veterans Affairs

Appendix IX: Comments from the Department of Defense

Appendix X: Comments from the Department of Homeland Security

Appendix XI: Comments from the Department of the Interior

Appendix XII: Comments from the Department of Health and Human Services

Appendix XIII: Comments from the Environmental Protection Agency

Appendix XIV: Comments from the General Services Administration

Appendix XV: Comments from the National Science Foundation

Appendix XVI: Comments from the Nuclear Regulatory Commission

Appendix XVII: Comments from the U.S. Agency for International Development

Appendix XVIII: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, individuals making contributions to this report included Dave Powner (director), Dave Hinchman (assistant director), Justin Booth (analyst-in-charge), Alexander Bennett, Chris Businsky, Nancy Glover, and Jonathan Wall.