Data Center Optimization: Continued Agency Actions Needed to Meet Goals and Address Prior Recommendations
Released: 2018-05-23T00:00:00

Letter

Background

The federal government’s increasing demand for IT has led to an increase in the number of federal data centers and a corresponding increase in operational costs. According to OMB, the federal government had 432 data centers in 1998, 2,094 in July 2010, and 9,995 in August 2016.

Operating such a large number of centers has been, and continues to be, a significant cost to the federal government. For example, in 2007, the Environmental Protection Agency (EPA) estimated that the electricity costs to operate federal servers and data centers across the government were about $450 million annually. According to the Department of Energy (Energy), a typical data center has 100 to 200 times the energy use intensity of a commercial building. In 2009, OMB reported that server utilization rates as low as 5 percent across the federal government’s estimated 150,000 servers were a factor driving the need to establish a coordinated, government-wide effort to improve the efficiency, performance, and environmental footprint of federal data center activities.

OMB and the Federal CIO Established FDCCI

Concerned about the size of the federal data center inventory and the potential to improve the efficiency, performance, and the environmental footprint of federal data center activities, OMB’s Federal CIO established FDCCI in February 2010. This initiative’s four high-level goals were to reduce the overall energy and real estate footprint of government data centers; reduce the cost of data center hardware, software, and operations; increase the overall IT security posture of the government; and shift IT investments to more efficient computing platforms and technologies.

In February 2010, OMB required all of the agencies participating in the FDCCI to submit a data center inventory and a consolidation plan. In October 2010, OMB also clarified the definition of a data center and noted that, for the purposes of FDCCI, a data center was to be defined as any room used for the purpose of processing or storing data that is larger than 500 square feet and meets stringent availability requirements. Under this definition, OMB reported that agencies had identified 2,094 data centers as of July 2010.

However, in 2011, the Federal CIO expanded the definition to include a facility of any size and OMB published its revised definition in March 2012. Based on the revised definition, OMB estimated that there were a total of 3,133 federal data centers in December 2011. In addition, its goal was to consolidate approximately 40 percent, or 1,253 of these data centers, for a savings of approximately $3 billion by the end of 2015. Figure 1 provides an example of data center server racks at the Social Security Administration’s (SSA) National Support Center.

The number of federal data centers reported by agencies has continued to grow since 2011. In March 2016, we reported that agencies had collectively identified a total of 10,584 data centers as of November 2015—an increase of about 7,500 data centers compared to OMB’s October 2011 estimate. According to the Federal CIO, the increase in the number of data centers was primarily due to the expanded definition of a data center and improved inventory reporting by the agencies.

Further, OMB placed greater emphasis on data center optimization when it issued memorandum M-13-09 in March 2013. Specifically, OMB stated that, to more effectively measure the efficiency of an agency’s data center assets, agencies would also be measured by the extent to which their primary data centers are optimized for total cost of ownership by incorporating metrics for data center energy, facility, labor, and storage, among other things. Subsequently, in May 2014, OMB issued memorandum M-14-08, which established a set of data center optimization metrics to measure agency progress, along with target values for each metric. All agencies were expected to achieve the target values by the end of fiscal year 2015.

IT Acquisition Reform Law Enhanced Data Center Consolidation and Optimization Efforts

Recognizing the importance of reforming the government-wide management of IT, Congress enacted FITARA in December 2014. Among other things, the law required agencies to:

Submit to OMB a comprehensive inventory of the data centers owned, operated, or maintained by or on behalf of the agency.

Submit, by the end of fiscal year 2016, a multi-year strategy to achieve the consolidation and optimization of the agency’s data centers. The strategy was to include performance metrics that were consistent with the government-wide data center consolidation and optimization metrics.

Report progress towards meeting government-wide data center consolidation and optimization metrics on a quarterly basis to OMB’s Administrator of the Office of Electronic Government.

In addition, according to FITARA, the Office of Electronic Government at OMB was to:

Establish metrics applicable to the consolidation and optimization of data centers (including server efficiency), ensure that agencies’ progress toward meeting government-wide data center consolidation and optimization metrics is made publicly available, review agencies’ inventories and strategies to determine whether they are comprehensive and complete, and monitor the implementation of each agency’s strategy.

Develop and make publicly available not later than December 19, 2015, a goal broken down by year, for the amount of planned cost savings and optimization improvements to be achieved through FDCCI and, for each year thereafter until October 1, 2020, compare reported cost savings and optimization improvements against those goals.

OMB Established DCOI

In August 2016, OMB issued memorandum M-16-19, which established DCOI and included guidance on how to implement the data center consolidation and optimization provisions of FITARA. Among other things, the guidance required agencies to consolidate inefficient infrastructure, optimize existing facilities, improve their security posture, and achieve cost savings. For example, each agency was required to maintain a complete inventory of all data center facilities owned, operated, or maintained by or on its behalf, and measure progress toward defined optimization performance metrics on a quarterly basis as part of its data center inventory submission.

OMB’s memorandum also directed each agency to develop a DCOI strategic plan that defined its data center strategy for fiscal years 2016 through 2018. Among other things, this strategy was to include a timeline for agency consolidation and optimization activities with an emphasis on cost savings and optimization performance benchmarks that the agency could achieve between fiscal years 2016 and 2018. For example, each agency was required to develop cost savings targets due to consolidation and optimization actions and report any realized cost savings. OMB required each agency to publicly post its DCOI strategic plan to its agency-owned digital strategy website by September 30, 2016, and to post subsequent strategic plan updates by April 14, 2017 and April 13, 2018.

Further, the memorandum stated that OMB was to maintain a public dashboard (referred to as the IT Dashboard) to display government-wide and agency-specific progress in areas such as planned and achieved data center closures, consolidation-related cost savings, and data center optimization performance information. In this regard, OMB began including data center consolidation and optimization progress information on the IT Dashboard in August 2016.

OMB’s memorandum also provided new guidance for the classification of a physical data center, expanding the definition of a data center. According to the revised definition, a room with at least one server that provides services (whether in a production, test, staging, development, or any other environment) should be considered a data center, while a room containing only print servers, routing equipment, switches, security devices (such as firewalls), or other telecommunication components, was not to be considered a data center.

In light of this new definition, OMB directed each agency to perform a comprehensive review of its data centers and maintain a complete and updated data center inventory. Further, OMB directed each agency to categorize each of its data centers as either a tiered data center or a non- tiered data center. OMB’s memorandum defined a tiered data center as one that uses each of the following: a separate physical space for IT infrastructure; an uninterruptible power supply; a dedicated cooling system or zone; and a backup power generator for a prolonged power outage.

According to the memorandum, all other data centers are to be considered non-tiered.

Moreover, OMB guidance included a series of performance metrics in the areas of data center closures, cost savings, and optimization progress.

Data center closures: Agencies are expected to close at least 25 percent of tiered data centers government-wide, excluding those approved as inter-agency shared services providers, by the end of fiscal year 2018. Further, agencies are to close at least 60 percent of non-tiered data centers government-wide by the end of fiscal year 2018. OMB’s guidance further notes that, in the long term, all agencies should continually strive to close all non-tiered data centers, noting that server rooms and closets pose security risks and management challenges and are an inefficient use of resources.

Cost savings: Agencies are expected to reduce government-wide annual costs attributable to physical data centers by at least 25 percent, resulting in savings of at least $2.7 billion for fiscal years 2016 through 2018.

Data center optimization: Agencies are expected to measure progress against a series of new data center performance metrics in the areas of server utilization, energy metering, power usage, facility utilization, and virtualization. Further, OMB’s guidance established target values for each metric that agencies are to achieve by the end of fiscal year 2018.

OMB’s guidance further noted that agency progress against these performance metrics is to be measured by OMB on a quarterly basis, using agencies’ data center inventory submissions and OMB-defined closures, cost savings, and optimization targets.

Agencies Have Taken Limited Action to Address GAO Recommendations from Prior Reports on Agencies’ Consolidation and Optimization Efforts

Since the enactment of FITARA in December 2014, we have annually reviewed and verified the quality and completeness of each covered agency’s inventory and DCOI strategy. We have also published reports documenting the findings from each of these reviews. In addition, we have examined and reported on agencies’ efforts to optimize their data centers, as well as the challenges encountered and successes achieved.

In a report that we issued in March 2016, we noted that agencies had reported significant data center closures—totaling more than 3,100 through fiscal year 2015—with the Departments of Agriculture (Agriculture), Defense (Defense), the Interior (Interior), and the Treasury (Treasury) accounting for 84 percent of the total. Although agencies fell short of OMB’s fiscal year 2015 consolidation goal, their plans identified about 2,100 additional centers planned for closure through fiscal year 2019.

Agencies also reported significant consolidation cost savings and avoidances—totaling about $2.8 billion through fiscal year 2015, and expected to increase to over $8.0 billion in future years. The Departments of Commerce (Commerce), Defense, Homeland Security (DHS), Transportation (Transportation), and the Treasury accounted for 96 percent of the total planned savings. However, we pointed out that many agencies lacked complete cost savings goals for the next several years despite having closures planned.

In addition, we reported that 22 agencies had made limited progress against OMB’s fiscal year 2015 data center optimization performance metrics, such as the utilization of data center facilities. Accordingly, we recommended that the agencies take actions to complete their cost savings targets and improve optimization progress. Of the 24 agencies to which we made recommendations, 14 agreed with our recommendations, 4 did not state whether they agreed or disagreed, and 6 stated that they had no comments. As of March 2018, 26 of the 32 recommendations from this report had yet to be fully addressed.

In May 2017, we reported that the agencies continued to report significant data center closures—totaling more than 4,300 through August 2016— with Agriculture, Defense, Interior, and the Treasury accounting for 84 percent of the total. The agencies’ plans for 2016 had identified more than 1,200 additional centers planned for closure through fiscal year 2019.

Agencies also reported significant consolidation and optimization cost savings and avoidances, which totaled about $2.3 billion through August 2016. However, reductions in the amount of achieved savings reported to OMB, particularly by the Treasury, resulted in a net decrease of more than $400 million in these savings, compared to amounts we previously reported in 2015.

Further, our report noted that, as of December 2016, agencies’ total planned cost savings of about $656 million were more than $3.3 billion less, compared to the amounts that we reported in 2015, and more than $2 billion less than OMB’s fiscal year 2018 cost savings goal of $2.7 billion. This reduction in planned savings was the result of eight agencies reporting less in planned cost savings and avoidances in their DCOI strategic plans compared to the savings amounts previously reported to us in November 2015. The reduction also reflected the absence of cost savings information for one agency (Defense) that did not submit its strategic plan in time for our review.

In addition, our May 2017 report identified weaknesses in agencies’ DCOI strategic plans. Of the 23 agencies that had submitted their strategic plans at the time of our review, 7 agencies—Agriculture, the Department of Education (Education), DHS, and the Department of Housing and Urban Development (HUD); the General Services Administration (GSA); the National Science Foundation (NSF); and the Office of Personnel Management (OPM)—had addressed all five required elements of a strategic plan, as identified by OMB (such as providing information related to data center closures and cost savings metrics). The remaining 16 agencies that submitted their plans either partially met or did not meet the requirements. We also pointed out that there were inconsistencies in the reporting of cost savings in the strategic plans of 11 agencies.

Given these findings, we recommended that OMB improve its oversight of agencies’ DCOI strategic plans and their reporting of cost savings and avoidances. We also recommended that 16 agencies and Defense (which did not submit a plan in time for our review) complete the missing elements in their strategic plans, and that 11 agencies ensure the reporting of consistent cost savings and avoidance information to OMB. Of the 25 agencies (including OMB) to which we made recommendations, 12 agreed with our recommendations, 2 disagreed, and 11 did not state whether they agreed or disagreed. As of March 2018, 29 of the 30 recommendations had not been fully addressed.

In a subsequent report that we issued in August 2017, we noted that 22 of the 24 agencies required to participate in the OMB DCOI collectively had reported limited progress against OMB’s fiscal year 2018 performance targets for the five optimization metrics. The 2 remaining agencies, Education and HUD, did not have agency-owned data centers and, therefore, did not have a basis to report on progress. Specifically, for each of the five targets, no more than 5 agencies reported that they had met or exceeded that specific target. This limited progress against OMB’s optimization targets was due, in part, to agencies not fully addressing our prior recommendations in this area.

In addition, we noted in the report that most agencies had not yet implemented automated monitoring tools to measure server utilization, as required by the end of fiscal year 2018. Specifically, 4 agencies reported that they had fully implemented such tools, 18 reported that they had not yet done so, and 2 did not have a basis to report on progress because they did not have agency-owned data centers.

We also noted that, although federal standards emphasize the need to establish plans to help ensure goals are met, none of the 18 agencies had fully documented plans for implementing automated monitoring tools. Accordingly, we recommended that OMB formally document a requirement for agencies to include plans, as part of existing OMB reporting mechanisms, to implement automated monitoring tools at their agency-owned data centers. We also recommended that the 18 agencies without fully documented plans take action, within existing OMB reporting mechanisms, to complete plans describing how they intend to achieve OMB’s requirement to implement automated monitoring tools at all agency-owned data centers by the end of fiscal year 2018. Of the 19 agencies (including OMB) to which we made recommendations, 10 agreed with our recommendations, 3 partially agreed, and 6 did not state whether they agreed or disagreed. As of March 2018, none of the 19 recommendations had been fully addressed.

Agencies Report Mixed Results Relative to Achieving OMB’s Data Center Closure Targets

As previously mentioned, in August 2016, OMB established a goal to close at least 25 percent of each agency’s tiered data centers and at least 60 percent of each agency’s non-tiered data centers by the end of fiscal year 2018. Related to doing so, agencies’ August 2016 inventories reported a total of 9,995 data centers and, in August 2017, 14 agencies reported an additional 2,067 facilities, for a total of 12,062 data centers. Based on this revised inventory, agencies will need to close 6,306 data centers (665 tiered and 5,641 non-tiered) to meet OMB’s goals by the end of fiscal year 2018.

Toward this end, the 24 agencies participating in DCOI collectively have made progress on their data center closure efforts. Specifically, as of August 2017, the agencies reported that they had closed 5,805 tiered and non-tiered centers (48 percent). Figure 2 provides a summary of the total number of federal data centers and closures reported from 1998 to August 2017.

Figure 3 provides a further breakdown of agencies’ data center inventories, as of August 2017, in terms of the total number of data centers that were closed, planned for closure, or not planned for closure.

Nevertheless, while the agencies collectively had made progress toward OMB’s closure goals, the 24 agencies’ individual reported progress and plans showed mixed results when compared with OMB’s goal for each agency to close at least 25 percent of tiered data centers and at least 60 percent of non-tiered centers. Specifically, as of August 2017, 13 agencies reported that they had already met the goal of closing 25 percent of their tiered data centers, another 4 agencies reported that they plan to meet the goal by the end of fiscal year 2018, and 5 agencies reported that they do not currently plan to meet the goal.

Further, as of August 2017, 7 agencies reported having already met the goal for closing 60 percent of their non-tiered centers, 6 agencies reported that they planned to meet the goal by the end of fiscal year 2018, and 10 agencies reported that they did not plan to meet the goal. Table 1 displays a breakdown of the number of reported tiered and non- tiered data centers and completed and planned closures by agency, as of August 2017.

As shown in the table, the reported closures of Agriculture (2,233 data centers), Defense (834), and Treasury (1,713) together accounted for 4,780 (or 82 percent) of the 5,805 data center closures. However, no other agency accounted for more than 187 (or 3 percent) of those closures. In addition, the remaining 1,416 planned closures are to be carried out across 20 agencies. Further, among the agencies, 7 reported that they do not plan to meet one of their tiered or non-tiered closure goals, and 4 reported that they do not plan to meet either of the goals by the end of fiscal year 2018.

Officials from these 11 agencies that do not plan to meet one or both of their goals provided various reasons for why they currently do not plan to do so. For example, officials in USAID’s Office of the Chief Information Officer stated that their agency had reported a number of server closets in overseas locations as non-tiered data centers to comply with OMB’s data center definition provided in its August 2016 guidance. The officials said that, as long as USAID maintains those locations, the agency needs the services provided in those server closets and will not be able to close them to meet OMB’s goal. However, the officials also said that the agency is exploring ways to replace the server closets using cloud services.

Several agencies that viewed their goals as unattainable indicated that they were seeking revised closure goals. Specifically, officials from Interior’s Office of the Chief Information Officer stated that a number of the department’s non-tiered data centers were either mission critical or not cost effective to close. Thus, the officials said Interior was working with OMB to establish a revised closure goal. Similarly, Transportation’s Director for IT Compliance stated that the department was working with OMB to establish a revised closure goal. The department reported having 186 tiered data centers in Federal Aviation Administration control towers that it believes should be excluded from its count of data centers when OMB sets the department’s goal for closures.

Further, our analysis determined that it may not always be realistic for an agency to meet OMB’s targets. For example, NSF, which does not have any tiered data centers, reported that it plans to close one (50 percent) of its two non-tiered data centers. However, the only way to meet OMB’s 60 percent threshold would be for the agency to close both of its non-tiered centers, which may not be an option, depending on the services provided by that one remaining center.

Although OMB’s stated time frame for closing data centers currently remains as the end of fiscal year 2018, the recent extension of FITARA’s data center consolidation and optimization provisions through fiscal year 2020 provides agencies additional time to work toward meeting OMB’s closure targets. However, in some cases, these efforts may require significant restructuring of an agency’s business operations, or, as reported by several agencies, a revision of OMB’s goals in consideration of the agencies’ specific needs.

Given that agencies had been working toward OMB’s DCOI goals for approximately one year as of August 2017, and because the extension of FITARA’s data center provisions pushes the sunset for these efforts out through fiscal year 2020, we are not making any related recommendations to those agencies that have not met the closure goals at this time. We plan to continue to monitor the agencies’ progress toward meeting the goals in our future work.

Agencies’ Planned Savings Are Not Expected to Meet OMB’s Goal

Since 2013, federal agencies have been required to report on data center cost savings. In this regard, OMB provided guidance regarding how agencies were to report cost savings and avoidances. Specifically, it required agencies to report both data center consolidation cost savings and avoidances, among other areas, as part of a quarterly data collection process known as the integrated data collection. FITARA also called for each agency to submit a multi-year strategy for achieving the consolidation and optimization of data centers that includes year-by-year calculations of investment and cost savings through fiscal year 2018, which has now been extended to 2020.

In addition, in August 2016, OMB M-16-19 provided guidance on how agencies should implement the requirements of FITARA. Specifically, agencies were to develop a strategic plan that included information on historical cost savings and avoidances due to data center consolidation and optimization through fiscal year 2015. This guidance stated that agency strategic plans were also to include year-by-year calculations of target and actual agency-wide spending and cost savings on data centers from fiscal years 2016 through 2018. Further, the guidance established a DCOI government-wide cost savings goal of $2.7 billion for all federal agencies to achieve from fiscal years 2016 through 2018. This overall goal is then broken down into agency-specific targets on the IT Dashboard.

As of August 2017, 20 agencies had reported through the integrated data collection that they had achieved $1.04 billion in cost savings for fiscal years 2016 and 2017, while 4 agencies reported that they had not achieved any savings. Further, the 20 agencies’ DCOI strategic plans identified an additional $0.58 billion, for a total of $1.62 billion in planned savings from fiscal years 2016 through 2018. Nevertheless, this total is about $1.12 billion below OMB’s goal of $2.7 billion for DCOI savings. Figure 4 provides a comparison of the 24 agencies’ total reported savings for fiscal years 2016 and 2017, and the planned savings through 2018, against OMB’s DCOI savings goal for fiscal years 2016 through 2018.

Among the 24 participating DCOI agencies that reported achieving about $1.04 billion in savings, Commerce ($594.28 million), Defense ($141.36 million), and DHS ($106.51 million) were responsible for approximately $842 million (about 81 percent) of that total. No other agency reported saving more than $54.40 million. Table 2 provides specific data related to each agency’s planned and achieved savings for fiscal years 2016 and 2017, as of August 2017.

In addition, the 24 agencies reported that they have planned an additional $0.58 billion in DCOI cost savings (for a total of $1.62 billion) through fiscal year 2018. However, as noted earlier, this total is approximately $1.12 billion below OMB’s $2.7 billion goal for the initiative. Table 3 provides a more detailed comparison between each agency’s planned savings, as reported in its DCOI strategic plan, and OMB’s agency- specific targets, as reported on the IT Dashboard.

As shown in table 3, 6 agencies identified planned savings that are expected to meet or exceed their OMB targets, while 2 agencies that did not have an OMB target also identified planned savings. In contrast, 12 agencies reported that they are not currently planning to meet their targets, and 4 agencies did not have a savings target and are not planning any savings.

These findings align with what we reported in March 2016, when we noted that 10 agencies had not established planned cost savings goals for fiscal years 2016 through 2018, even though they had closures planned during that time period. Accordingly, in that report, we recommended that these agencies complete their planned data center cost savings targets for fiscal years 2016 through 2018. Most of the agencies agreed with the recommendations or had no comments. Nonetheless, agencies continued to be challenged in identifying and reporting their cost savings. As of August 2017, 5 of the agencies had implemented our recommendations in this area.

In the absence of consistent and full reporting of fiscal years 2016 through 2018 planned savings in agencies’ DCOI strategic plans, as required by FITARA and OMB, agencies’ total planned savings will likely continue to be understated. With less than a year for agencies to meet OMB’s current planned savings targets, we are re-emphasizing the need for agencies to implement our prior recommendations related to establishing and meeting their planned data center cost savings targets.

Agencies Continue Reporting Limited Progress against OMB’s Data Center Optimization Metrics

FITARA required OMB to establish metrics to measure the optimization of data centers, including server efficiency, and ensure that agencies’ progress toward meeting the metrics is made public. Pursuant to FITARA, OMB’s August 2016 memorandum established a set of five data center optimization metrics intended to measure agencies’ progress in the areas of server utilization and automated monitoring, energy metering, power usage effectiveness, facility utilization, and virtualization. According to OMB, while the server utilization and automated monitoring metric applies to agency-owned tiered and non-tiered data centers, the four remaining metrics apply only to agency-owned tiered centers.

OMB’s memorandum also established a target value for each of the five metrics, which agencies are expected to achieve by the end of fiscal year 2018. OMB measures agencies’ progress against the optimization targets using the agencies’ quarterly data center inventory submission and publicly reports this progress information on its IT Dashboard. Table 4 provides a description of the five data center optimization metrics and target values.

As of August 2017, 22 of the 24 DCOI agencies continued to report limited progress in meeting OMB’s fiscal year 2018 data center optimization targets identified on the IT Dashboard. As noted earlier, the remaining 2 agencies—Education and HUD—reported that they did not have any agency-owned data centers in their inventory and, therefore, did not have a basis to measure and report optimization progress.

With regard to the data center optimization targets, the most progress was reported for the power usage effectiveness and virtualization metrics, with 5 and 6 agencies, respectively, reporting that they had met OMB’s targets. However, only 3 agencies or fewer reported meeting the energy metering, facility utilization, and server utilization and automated monitoring metrics. Figure 5 summarizes the 24 agencies’ progress in meeting each optimization target, as of August 2017.

As of August 2017, SSA and EPA reported the most progress among the 22 agencies with a basis to report against OMB’s metrics targets— meeting 4 and 3 targets, respectively. Six agencies met either one or two targets, and 14 agencies reported meeting none of the targets.

Further, of the 22 agencies, 9 were not able to report any progress against either the server utilization metric or power usage effectiveness metric, or both, because their data centers lacked the required monitoring tools to measure progress in these areas. OMB began requiring the implementation of these monitoring tools in August 2016; however, as of August 2017, these 9 agencies had not reported that they had implemented the tools at any data centers. The remaining 13 agencies reported that they had implemented the tools in at least one data center. Table 5 depicts the agencies and whether they met or did not meet each OMB target.

Agencies’ limited progress against OMB’s optimization targets is due, in part, to not fully addressing our prior recommendations in this area. As discussed earlier, in March 2016, we reported on weaknesses in agencies’ data center optimization efforts, including that 22 agencies did not meet OMB’s fiscal year 2015 optimization targets. We noted that this was partially due to the agencies facing challenges in optimizing their data centers, including their decentralized organizational structures that made consolidation and optimization difficult, and competing priorities for resources. In addition, consolidating certain data centers was problematic because the volume or type of information involved required the data center to be close in proximity to the users. Accordingly, we recommended that the agencies take action to improve optimization progress, to include addressing any identified challenges. Most agencies agreed with our recommendations or had no comments.

In response to our recommendation, 19 of the 22 agencies submitted documentation to us that described steps they intended to take to improve their data center optimization efforts. The planned steps included completion dates ranging between April 2016 and September 2019. Among the steps described by the agencies was developing internal scorecards to track and report on optimization progress, including progress at their component agencies, and launching more aggressive efforts to optimize data centers using virtualization and cloud computing solutions. However, as of February 2018, only 1 of the 22 agencies (Education) had fully addressed our recommendation.

Agencies’ Planned Optimization Progress Will Not Meet OMB’s Fiscal Year 2018 Targets

In addition to reporting current optimization progress on the IT Dashboard, OMB requires agencies’ DCOI strategic plans to include, among other things, planned performance levels for fiscal years 2017 and 2018 for each optimization metric.

However, according to the 24 agencies’ DCOI strategic plan information as of August 2017, most are not planning to meet OMB’s optimization targets by the end of fiscal year 2018. More specifically, of the 24 agencies, only 4—Commerce, EPA, NSF, and USAID—reported plans to fully meet their applicable targets by the end of fiscal year 2018. Of the remaining agencies, 14 reported plans to meet some, but not all, of the targets; 4 reported that they do not plan to meet any targets; and—as already discussed—Education and HUD do not have a basis to report planned optimization milestones because they do not report having any agency-owned data centers. Figure 6 summarizes agencies’ progress, as of August 2017, in meeting OMB’s optimization targets and planned progress to be achieved by September 2018.

The limited progress made by agencies in optimizing their data centers, combined with the lack of established plans to improve progress, makes it unclear whether agencies will be able to achieve OMB’s optimization targets by the end of fiscal year 2018. Considering that OMB is expecting at least $2.7 billion in cost savings from agencies’ optimization efforts, the ability of agencies to meet the optimization targets will be critical to meeting this savings goal. However, only four agencies are planning to meet all of their applicable targets. If the remaining agencies take steps to implement the prior recommendations we have made in this area, it should increase the likelihood that DCOI can achieve the expected benefits of optimization and the resulting cost savings.

Agency Comments and Our Evaluation

We requested comments on a draft of this report from OMB and the 24 other agencies that we reviewed. Of these, 5 agencies indicated that they agreed with our report and 20 acknowledged receiving the draft, but did not state whether they agreed or disagreed with the report. Agencies also offered various comments in support of the DCOI effort and actions taken to improve performance. Additionally, multiple agencies provided technical comments, which we have incorporated, as appropriate.

The following five agencies agreed with our report: In emails received from Agriculture, Energy, VA, and NRC, the agencies agreed with the findings in the draft report. In written comments, SSA agreed with the draft report’s characterization of the agency’s DCOI efforts. SSA’s comments are reprinted in appendix II.

In addition, the following 20 agencies did not state whether they agreed or disagreed with the report and offered other comments: In written comments, HUD and Transportation did not agree or disagree with the draft report. The departments’ comments are reprinted in appendices III and IV, respectively.

In emails received from Commerce, Defense, Education, HHS, DHS, Justice, Labor, State, Treasury, EPA, NASA, NSF, OPM, SBA, and USAID, the agencies did not agree or disagree with the draft report.

In written comments, Interior did not agree or disagree with the draft report’s findings.

In addition, Interior provided a technical comment related to our calculation of the department’s achieved optimization savings for fiscal years 2016 and 2017. Specifically, the department provided updated savings figures for those years, which were based on Interior’s February 2018 quarterly submission to OMB.

However, the effective date of the updated data (February 2018) is outside the scope of our review, which relied solely on data reported in August 2017, as detailed in appendix I. While we recognize the department’s efforts to ensure that its past reporting is updated and as accurate as possible, our report only presents data as reported by agencies in August 2017. Consequently, we believe that the achieved cost savings for Interior accurately reflect what the department reported at the time of our review. The department’s comments are reprinted in appendix V.

In an e-mail received on March 27, 2018, GSA did not agree or disagree with the draft report’s findings, and provided comments questioning the methodology we used to determine the number of data center closures. Specifically, GSA stated that, by including facilities that are now closed in our reporting of the federal government’s inventory of 12,062 data centers, our report gives the impression that the government is currently operating 12,062 open data centers. GSA also asserted that we should include in our counts facilities identified as using cloud computing providers.

In our report, we consistently state that the 12,062 data centers reflect the overall count of federal facilities identified since the launch of FDCCI in 2010. Further, we clearly identify the portion of the overall count of data centers that have since closed, or that are planned to close. By doing so, we provide a perspective of consolidation progress against the overall inventory over the past 8 years.

Further, in regard to facilities using cloud providers, OMB’s August 2016 guidance specifically states that such facilities are not to be considered data centers. As such, we did not include them in our totals. Accordingly, we maintain that our methodology is reasonable and continue to believe that our report accurately reports the status of federal data center consolidation efforts.

GSA also suggested that we confirm with OMB our statement that the FITARA Enhancement Act of 2017’s extension of the sunset provides agencies with an additional 2 years to accomplish the goals of DCOI. It is true that OMB has not issued guidance that extends the existing August 2018 DCOI deadlines. However, it is appropriate to note that the 2017 law extends the DCOI sunset date for the data center requirements that govern agencies.

Finally, GSA expressed concern that the data on optimization metrics that we drew from the IT Dashboard were not representative of the data provided by agencies in their August 2017 quarterly submissions to OMB. Further, GSA noted that such a selection of mixed data sources may present an incomplete and inaccurate picture and recommended that we note the date on which we accessed the IT Dashboard data, note that those data are frequently updated, and specify the date of other sources of data that we used for our analysis.

In this report, we present an analysis of agencies’ progress against OMB’s data center optimization metrics, using data that were taken from the IT Dashboard. These data are posted to the Dashboard after being automatically calculated from agencies’ quarterly submissions to OMB. For the purposes of this report, we pulled the data from the Dashboard a week after agencies’ August 2017 submissions were due to OMB. We then confirmed with the agencies that the data we collected were consistent with their August 2017 submissions to OMB and we labeled the data as being effective as of August 2017.

We also analyzed other data from the agencies’ August 2017 quarterly submissions, such as the status of data center closures and associated cost savings, and showed the effective date of our analysis as being August 2017. Based on these actions, which are also discussed in the description of the scope and methodology of our audit work found in app. I, we believe that our presentation of the data on optimization metrics is accurate, appropriately labeled, and correctly reflects the status of agency efforts at a specific moment in time.

In an e-mail received on April 18, 2018, OMB did not agree or disagree with our report, but offered several comments on our findings. Specifically, OMB noted that some agencies may be reporting their planned savings incorrectly in their DCOI strategic plans in that agencies may be reporting annual savings figures instead of the required cumulative figures. OMB further described plans to update the IT Dashboard in the near future to more accurately reflect planned DCOI savings and added that, as a result, the data in our report likely will not match the data OMB intends to publish.

In conducting our analysis, on two occasions, we requested and received agencies’ validation of the results of our analysis of their planned cost savings. This process resulted in minor technical changes to some agencies’ data. We believe that our continued efforts to validate these data provide reasonable assurance as to the accuracy of the agency reported information that we analyzed. The implementation of OMB’s proposed changes to the IT Dashboard should provide yet another tool that can be used to improve how agencies report their cost savings—an important measure of DCOI’s success.

Additionally, the comments noted that GAO and OMB use a different basis to calculate agencies’ data center closure targets, with OMB using a baseline inventory from the beginning of DCOI in 2016, which does not recognize changes in agencies’ inventories since that point. OMB asserted that, because GAO’s calculations account for changes in agencies’ inventories since the beginning of DCOI, OMB’s targets differ from GAO’s calculations.

We recognize the difference in approach for calculating data center closures that we used, as compared to that used by OMB. As detailed in our report, when DCOI was launched in August 2016, agencies reported an overall inventory of 9,995 data centers. One year later, in August 2017, agencies reported more than 2,000 additional facilities, for a total of 12,062 centers.

Because OMB’s closure targets required agencies to close a certain percentage of their data centers, basing agency goals on an outdated inventory (that did not include the additional facilities) does not give a true picture of progress towards consolidation of data centers. Conversely, our methodology (detailed in app. I) takes into account inventory growth and uses the same percentage-based closure goals defined in OMB’s guidance. Our methodology presents a more accurate status of progress based on the growth of the inventory over time. Consequently, we believe that our methodology allows us to present a reasonable status of agencies’ progress against OMB’s goals.

We are sending copies of this report to interested congressional committees, the Director of OMB, the secretaries and heads of the departments and agencies addressed in this report, and other interested parties. In addition, the report will be available at no charge on GAO’s website at http://www.gao.gov.

If you or your staffs have any questions about this report, please contact me at (202) 512-9286 or pownerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix VI.

Appendix I: Objectives, Scope, and Methodology

Our objectives for this engagement were to: (1) review agencies’ data center closures to date and plans for further closures, (2) evaluate agencies’ progress in achieving data center consolidation savings and describe plans for future savings, and (3) assess agencies’ progress against the Office of Management and Budget’s (OMB) data center optimization targets.

To address the first objective, we obtained and analyzed August 2017 data center inventory documentation from the 24 departments and agencies (agencies) that participate in OMB’s Data Center Optimization Initiative (DCOI). To determine data center closures to date, we totaled agency reported closures from fiscal year 2010 through August 2017 and, to identify future closures, we totaled agency reported planned closures through fiscal year 2019. We also compared agencies’ completed and planned closures to OMB’s fiscal year 2018 consolidation goals, as documented in its August 2016 memorandum (M-16-19).

To verify the quality, completeness, and reliability of the agencies’ data center inventories, we compared information on completed and planned data center closures to similar information reported on OMB’s Information Technology (IT) Dashboard—a public website that provides information on federal agencies’ major IT investments. We also checked for missing data and other errors, such as missing closure status information. Further, we obtained written responses from agency officials regarding actions taken to ensure the reliability of their inventory data, and discussed any discrepancies or potential errors identified to determine the causes or to request additional information. We determined that the data were sufficiently complete and reliable to report on agencies’ consolidation progress and planned closures.

For the second objective, we obtained and analyzed cost savings and avoidance documentation from the 24 DCOI agencies. This documentation is required by OMB’s March 2013 and August 2016 memorandums and included the agencies’ quarterly reports of cost savings and avoidances posted to their digital services websites and their DCOI strategic plans. To determine cost savings achieved, we totaled agencies’ reported savings and avoidances from the start of fiscal years 2012 through August 2017, as found in the August 2017 quarterly reports posted to the agencies’ digital services websites. To identify future planned savings, we totaled the agencies’ projected savings and avoidances from fiscal years 2016 through 2018, as reported in their DCOI strategic plans.

To assess the quality, completeness, and reliability of each agency’s data center consolidation cost savings information, we used the latest version of each agency’s update of the August 2017 quarterly cost savings report and DCOI strategic plan. We also reviewed the quarterly reports and DCOI strategic plans for missing data and other errors, such as missing cost-savings information. In addition, we compared agencies cost savings and avoidances with data from our most recent data center consolidation report. Further, we obtained written responses from agency officials regarding the steps taken to ensure the accuracy and reliability of their cost savings data. As a result, we determined that the data were sufficiently complete and reliable to report on agencies data center consolidation cost-savings information.

For our third objective, we analyzed the August 2017 data center optimization progress information of the 24 DCOI agencies. This progress information was obtained from the IT Dashboard—an OMB public website that provides information on federal agencies’ major IT investments. We then compared the agencies’ optimization progress information against OMB’s fiscal year 2018 optimization targets, as documented in its August 2016 memorandum. Although OMB’s memorandum establishes a single optimization target value for the server utilization and automated monitoring metric, the IT Dashboard displays agencies’ progress for tiered and non-tiered data centers separately. To report consistently with OMB’s implementation memorandum, we combined the progress information for tiered and non-tiered data centers into a single assessment in this report.

To assess the reliability of agencies’ optimization progress information on OMB’s IT Dashboard, we reviewed the information for errors or missing data, such as progress information that was not available for certain metrics. We also compared agencies’ optimization progress information across multiple reporting quarters to identify any inconsistencies in agencies’ reported progress. We discussed with staff from OMB’s Office of the Federal Chief Information Officer any discrepancies or potential errors identified to determine the causes.

In addition, we interviewed OMB officials to obtain additional information regarding the steps taken to ensure the reliability of and validate the optimization data on the Dashboard. Moreover, we obtained written responses from agency officials regarding the steps taken to ensure the accuracy and reliability of the reported optimization progress. We discussed with agency officials any discrepancies or potential errors identified during our reviews to determine the causes or request additional information. We determined that the data were sufficiently reliable to report on agencies’ optimization progress.

To assess the reliability of the DCOI strategic plans, we reviewed agencies’ documentation to identify any missing data or errors. We also compared the planned data center optimization milestones in agencies’ documentation against current optimization progress information obtained from the IT Dashboard. In addition, we discussed with agency officials any discrepancies or potential errors identified during our reviews of the DCOI strategic plans to determine the causes or request additional information. As a result of these efforts, we determined that the agencies’ strategic plan information was sufficiently reliable for reporting on plans to meet or not meet OMB’s fiscal year 2018 optimization targets.

We conducted this performance audit from July 2017 to May 2018 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Social Security Administration

Appendix III: Comments from the Department of Housing and Urban Development

Appendix IV: Comments from the Department of Transportation

Appendix V: Comments from the Department of the Interior

Appendix VI: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, individuals making contributions to this report included Dave Hinchman (Assistant Director), Justin Booth (Analyst-in-Charge), Chris Businsky, Nancy Glover, Linda Kochersberger, and Jonathan Wall.