Data Center Optimization: Agencies Need to Address Challenges and Improve Progress to Achieve Cost Savings Goal
Released: 2017-09-06T00:00:00

Letter

Background

The federal government’s increasing demand for IT has led to an increase in the number of federal data centers and a corresponding increase in operational costs. According to OMB, the federal government reported 432 data centers in 1998, 2,094 in July 2010, and 9,995 in August 2016. Operating such a large number of centers has been and continues to be a significant cost to the federal government, including costs for hardware, software, real estate, and cooling. For example, in 2007, the Environmental Protection Agency (EPA) estimated that the electricity costs to operate federal servers and data centers across the government were about $450 million annually. According to the Department of Energy (Energy), a typical data center has 100 to 200 times the energy use intensity of a commercial building. In 2009, OMB reported that server utilization rates as low as 5 percent across the federal government’s estimated 150,000 servers were a factor driving the need to establish a coordinated, government-wide effort to improve the efficiency, performance, and environmental footprint of federal data center activities.

OMB and the Federal CIO Established FDCCI

Concerned about the size of the federal data center inventory and the potential to improve the efficiency, performance, and the environmental footprint of federal data center activities, OMB, under the direction of the Federal CIO, established FDCCI in February 2010. This initiative’s four high-level goals were to promote the use of “green IT” by reducing the overall energy and real estate footprint of government data centers; reduce the cost of data center hardware, software, and operations; increase the overall IT security posture of the government; and shift IT investments to more efficient computing platforms and technologies.

As part of the initiative, OMB required the 24 agencies to identify a data center consolidation program manager to lead the agency’s consolidation efforts. In addition, agencies were required to submit an asset inventory baseline and other documents that would result in a plan for consolidating their data centers. The asset inventory baseline was to contain detailed information on each data center and identify the consolidation approach to be taken for each one. It would serve as the foundation for developing the final data center consolidation plan. The data center consolidation plan would serve as a technical road map and approach for achieving the targets for infrastructure utilization, energy efficiency, and cost efficiency.

In October 2010, OMB reported that all of the agencies had submitted an inventory and plan. OMB also clarified the definition of a data center and noted that, for the purposes of FDCCI, a data center is defined as any room used for the purpose of processing or storing data that is larger than 500 square feet and meets stringent availability requirements. Under this definition, OMB reported that agencies had identified 2,094 data centers as of July 2010. “…a data center is…a closet, room, floor, or building for the storage, management, and dissemination of data and information and computer systems and associated components, such as database, application, and storage systems and data stores [excluding facilities exclusively devoted to communications and network equipment (e.g., telephone exchanges and telecommunications rooms)]. A data center generally includes redundant or backup power supplies, redundant data communications connections, environmental controls…and special security devices housed in leased, owned, collocated, or stand-alone facilities.”

Under the new definition, OMB estimated that there were a total of 3,133 federal data centers in December 2011, and its goal was to consolidate approximately 40 percent, or 1,253 data centers, for a savings of approximately $3 billion by the end of 2015. See figure 1 for an example of an image of data center server racks at the Social Security Administration’s (SSA) National Support Center.

In March 2012, OMB launched the PortfolioStat initiative, which requires agencies to conduct an annual agency-wide IT portfolio review to, among other things, reduce commodity IT spending and demonstrate how its IT investments align with the agency’s mission and business functions. PortfolioStat is designed to assist agencies in assessing the current maturity of their IT portfolio management process, make decisions on eliminating duplication, and move to shared solutions in order to maximize the return on IT investments across the portfolio.

Subsequently, in March 2013, OMB issued a memorandum that documented the integration of FDCCI with PortfolioStat and stated that agencies should focus on an enterprise-wide approach to address commodity IT (including data centers) in a comprehensive manner. The memorandum also discussed consolidating previously collected IT-related plans, reports, and data submissions. For example, agencies were no longer required to submit the data center consolidation plans previously required in 2012. However, OMB required agencies to update their data center inventories and report on consolidation progress at the end of every quarter.

OMB’s 2013 memorandum also increased the focus on optimizing the performance of federal data centers. Specifically, OMB stated that, to more effectively measure the efficiency of an agency’s data center assets, agencies would also be measured by the extent to which their primary data centers were optimized for total cost of ownership by incorporating metrics for data center energy, facility, labor, and storage, among other things. Subsequently, in May 2014, OMB issued memorandum M-14-08, which established a set of data center optimization metrics to measure agency progress. In addition, OMB established target values that agencies were expected to achieve by the end of fiscal year 2015.

IT Acquisition Reform Law Enhanced Data Center Consolidation and Optimization Efforts

Recognizing the importance of reforming the government-wide management of IT, Congress enacted FITARA in December 2014. Among other things, the law includes a number of requirements related to federal data center consolidation and optimization:

Agencies shall submit to OMB a comprehensive inventory of the data centers owned, operated, or maintained by or on behalf of the agency.

Agencies shall submit a multi-year strategy to achieve the consolidation and optimization of the agency’s data centers no later than the end of fiscal year 2016. This strategy should include, for example, performance metrics that are consistent with the government-wide data center consolidation and optimization metrics.

On a quarterly basis, agencies shall report to OMB’s Administrator of the Office of Electronic Government on progress towards meeting government-wide data center consolidation and optimization metrics.

OMB’s Administrator of the Office of Electronic Government shall establish metrics applicable to the consolidation and optimization of data centers (including server efficiency), ensure that agencies’ progress toward meeting government-wide data center consolidation and optimization metrics is made publicly available, review agencies’ inventories and strategies to determine whether they are comprehensive and complete, and monitor the implementation of each agency’s strategy.

Not later than December 19, 2015, OMB’s Administrator of the Office of Electronic Government shall develop and make publicly available, a goal, broken down by year, for the amount of planned cost savings and optimization improvements achieved through FDCCI and, for each year thereafter through October 1, 2018, compare reported cost savings and optimization improvements against those goals.

The law’s data center consolidation and optimization provisions expire on October 1, 2018.

In June 2015, OMB memorandum M-15-14 provided guidance for implementing FITARA and related IT management practices. OMB’s guidance includes several actions that agencies are to take to establish a basic set of roles and responsibilities (referred to as the “common baseline”) for CIOs and other senior agency officials that are needed to implement the authorities described in the law. For example, agencies are to conduct a self-assessment to identify where they conform to the common baseline and where they deviate. OMB guidance also requires agencies to annually update their self-assessments and report their progress in reaching FITARA implementation milestones.

OMB Established DCOI

In August 2016, OMB issued a memorandum that established DCOI and included guidance on how to implement the data center consolidation and optimization provisions of FITARA. Among other things, the guidance requires agencies to consolidate inefficient infrastructure, optimize existing facilities, improve their security posture, and achieve cost savings. For example, agencies are required to maintain a complete inventory of all data center facilities owned, operated, or maintained by or on behalf of the agencies and measure progress toward defined optimization performance metrics on a quarterly basis as part of their data center inventory submissions.

OMB’s August 2016 memorandum also revised the definition of a physical data center to include any room with at least one server that provides services (such as testing and development). Further, OMB’s guidance directed agencies to categorize their data centers as either a tiered data center or a non-tiered data center. OMB guidance defines a tiered data center as one that uses each of the following: a separate physical space for IT infrastructure, an uninterruptible power supply, a dedicated cooling system or zone, and a backup power generator for a prolonged power outage. According to OMB, all other data centers shall be considered non-tiered.

Regarding data center optimization planning, the memorandum directs agencies to develop DCOI strategic plans that define their data center strategies for fiscal years 2016 through 2018. Among other things, this strategy is to include a timeline for agency consolidation and optimization activities with an emphasis on cost savings and optimization performance benchmarks the agency can achieve between fiscal years 2016 and 2018. For example, agencies are required to establish planned data center optimization milestones and report on progress toward achieving those milestones in their strategic plans. OMB required agencies to publicly post the plans to their agency-owned digital strategy websites by September 30, 2016, and to post subsequent strategic plan updates by April 14, 2017, and April 13, 2018.

OMB also directed agencies to update their publicly available FITARA implementation milestone information to identify, at a minimum, five milestones per fiscal year to be achieved through DCOI. According to OMB, the DCOI milestones are expected to be updated quarterly as progress is achieved and are to be reviewed in quarterly meetings with OMB staff.

Further, the memorandum states that OMB will report government-wide and agency-specific progress on the IT Dashboard—a public website that provides detailed information on major IT investments. According to OMB, this progress information is to include planned and achieved data center closures, consolidation-related costs savings, and data center optimization performance information. In this regard, OMB began including data center consolidation and optimization progress information on the Dashboard in August 2016.

Moreover, OMB guidance includes a series of performance metrics in the areas of data center closures, cost savings, and optimization progress.

Data center closures: Agencies are expected to close at least 25 percent of tiered data centers government-wide, excluding those approved as inter-agency shared services providers, by the end of fiscal year 2018. Further, agencies are to close at least 60 percent of non-tiered data centers government-wide by the end of fiscal year 2018. OMB’s guidance further notes that, in the long term, all agencies should continually strive to close all non-tiered data centers, noting that server rooms and closets pose security risks and management challenges and are an inefficient use of resources.

Cost savings: Agencies are expected to reduce government-wide annual costs attributable to physical data centers by at least 25 percent, resulting in savings of at least $2.7 billion, by the end of fiscal year 2018.

Data center optimization: Agencies are expected to measure progress against a series of new data center performance metrics in the areas of server utilization, energy metering, power usage, facility utilization, and virtualization. Further, OMB’s guidance establishes target values for each metric that agencies are to achieve by fiscal year 2018.

To improve the measurement of data center optimization progress, OMB’s memorandum directs agencies to replace the manual collection and reporting of systems, software, and hardware inventory housed within data centers with automated monitoring, inventory, and management tools (e.g., data center infrastructure management) by the end of fiscal year 2018. According to OMB, these data center tools (henceforth referred to as “automated monitoring tools”) are to provide the capability to, at a minimum, measure progress toward server utilization and virtualization metrics. While implementation of automated monitoring tools is not required to be completed until the end of fiscal year 2018, the memorandum strongly encourages agencies to implement them throughout their data centers immediately.

While OMB is primarily responsible for DCOI, its August 2016 memorandum designated the General Services Administration’s (GSA) Office of Government-wide Policy as a managing partner of the federal government data center line of business and data center shared services. More specifically, OMB’s memorandum states that this office is responsible for, among other things, providing guidance on technology advancements, innovation, cybersecurity, and best practices to data center providers and consumers of data center services. Further, the memorandum states that the office is responsible for assisting with creating and maintaining an inventory of acquisition tools and products related to data center optimization, including procurement vehicles for the acquisition of automated monitoring tools.

GAO Has Reported and Testified on Issues Related to Consolidating and Optimizing Data Centers

From July 2011 through May 2017, we issued a number of reports and testified on agency efforts to consolidate and optimize federal data centers and achieve cost savings. For example, in September 2014, we reported that, while agencies had made progress on their consolidation efforts, the total number of data centers reported by agencies had continued to grow since 2011 as a result of OMB’s expanded definition and improved inventory reporting. More specifically, we determined that agencies had collectively reported 9,658 data centers in their inventories—an increase of about 6,500 compared to OMB’s previous estimate from December 2011. We noted that agencies had plans to close about 3,700 data centers by September 2015.

We also reported that 19 of the 24 FDCCI agencies had collectively reported achieving an estimated $1.1 billion in cost savings for fiscal years 2011 through 2013, and that, by 2017, that figure was estimated to rise to about $5.3 billion. However, we pointed out that planned savings may be higher because 6 agencies—the Departments of Health and Human Services (HHS), Interior (Interior), Justice (Justice), and Labor (Labor), GSA, and the National Aeronautics and Space Administration (NASA)—that reported closing as many as 67 data centers had also reported limited or no savings.

In addition, our 2014 report noted that 11 of the 21 agencies with planned cost savings had underreported their fiscal years 2012 through 2015 figures to OMB by approximately $2.2 billion. While several agencies noted communication issues as the reason for underreporting, others did not provide a reason. We concluded that, until agencies fully report their savings, the $5.3 billion in total savings would be understated. Further, we reported that OMB’s May 2014 data center optimization metrics did not address server utilization, even though OMB reported this to be as low as 5 percent across the federal government in 2009. We noted that, without this metric, OMB may lack important information on agencies’ progress. As a result, we recommended that it implement a metric for server utilization and assist six agencies in reporting their consolidation cost savings; we also recommended that agencies fully report their consolidation cost savings. OMB and the agencies to which we made recommendations generally agreed with them. OMB subsequently established a metric to measure agencies’ server utilization progress in its August 2016 memorandum.

In March 2016, we reported that agencies had continued to make progress in their data center consolidation efforts. Specifically, we noted that agencies had reported closing 3,125 of the 10,584 total data centers as of November 2015. We further noted that 19 of the 24 agencies had reported achieving an estimated $2.8 billion in cost savings and avoidances from their data center consolidation and optimization effort for fiscal years 2011 through 2015. Agencies were also planning an additional $5.4 billion in cost savings and avoidances, for a total of approximately $8.2 billion, through fiscal year 2019. However, we noted that planned savings may be higher because 10 agencies that reported planned closures from fiscal years 2016 through 2018 had not fully developed their cost savings goals for these fiscal years.

In addition, we reported that 22 agencies had made limited progress against OMB’s fiscal year 2015 data center optimization performance metrics, such as the utilization of data center facilities. Accordingly, we recommended that the agencies take actions to complete their cost savings targets and improve optimization progress. Most agencies agreed with the recommendations or had no comments.

Finally, in May 2017, we reported that agencies continued to consolidate their data centers, including closing 4,388 of the 9,995 total data centers as of August 2016. Figure 2 provides a summary of the total number of data centers and closures reported from 1998 through August 2016.

However, we pointed out that agency progress in achieving savings had slowed and planned goals had been reduced. Specifically, 18 of the 24 agencies had reported achieving an estimated $2.3 billion in cost savings and avoidances from their data center consolidation and optimization efforts from the start of fiscal year 2012 to August 2016, which was about $451 million less than the total amount of achieved cost savings and avoidances that agencies reported to us in November 2015. In addition, agencies’ total planned cost savings of about $633 million were more than $3.4 billion less compared to the amounts that agencies reported to us in November 2015, and more than $2.1 billion less than OMB’s fiscal year 2018 cost savings goal of $2.7 billion.

Our May 2017 report also identified weaknesses in agencies’ DCOI strategic plans. Of the 23 agencies that submitted their strategic plans at the time of our review, 7—the Departments of Agriculture (Agriculture), Education (Education), Homeland Security (DHS), and Housing and Urban Development (HUD); GSA; the National Science Foundation (NSF); and the Office of Personnel Management (OPM)—had addressed all five required elements of a strategic plan, as identified by OMB (such as providing information related to data center closures and cost savings metrics). The remaining 16 agencies either partially met or did not meet the requirements. We also pointed out that there were inconsistencies in the reporting of cost savings in the strategic plans of 11 agencies.

We concluded that, until agencies address the weaknesses in their DCOI strategic plans, they may be challenged in implementing the data center consolidation and optimization provisions of FITARA. Accordingly, we recommended that OMB improve its oversight of agencies’ DCOI strategic plans and their reporting of cost savings and avoidances. We also recommended that 17 agencies complete the missing elements in their strategic plans and that 11 agencies ensure the reporting of consistent cost savings and avoidance information to OMB. Twelve agencies agreed with our recommendations, 2 disagreed, and 11 did not state whether they agreed or disagreed. The 2 agencies that disagreed— HUD and the Nuclear Regulatory Commission (NRC)—asserted that they had submitted complete strategic plans. After further review, we agreed that HUD had provided a complete plan and removed our recommendation. However, we determined that NRC’s plan was still incomplete and maintained that our recommendation was appropriate.

Agencies Are Reporting Limited Progress against OMB’s Data Center Optimization Targets

As mentioned earlier, FITARA required OMB to establish metrics to measure the optimization of data centers, including server efficiency, and ensure that agencies’ progress toward meeting the metrics is made publicly available. Pursuant to FITARA, OMB’s August 2016 memorandum established a set of five data center optimization metrics intended to measure agency’s progress in the areas of server utilization and automated monitoring, energy metering, power usage effectiveness, facility utilization, and virtualization. According to OMB, the server utilization and automated monitoring metric applies to agency-owned tiered and non-tiered data centers, while the four remaining metrics apply to agency-owned tiered centers only.

OMB’s memorandum also established a target value for each of the five metrics, which agencies are expected to achieve by the end of fiscal year 2018. OMB measures agencies’ progress against the optimization targets using the agencies’ quarterly data center inventory submission and publicly reports this progress information on its Dashboard. Table 1 provides a description of the data center optimization metrics and target values that agencies are expected to achieve by the end of fiscal year 2018.

As of February 2017, 22 of the 24 DCOI agencies reported limited progress against OMB’s fiscal year 2018 data center optimization targets on the Dashboard. The remaining 2 agencies—Education and HUD— reported that they did not have any agency-owned data centers in their inventory and, therefore, did not have a basis to measure and report optimization progress.

With regard to the data center optimization targets, the most progress was reported for the power usage effectiveness and virtualization metrics, with 5 agencies reporting that they had met OMB’s targets. However, 2 agencies or less reported meeting the target for energy metering, facility utilization, and server utilization and automated monitoring. Figure 3 summarizes the 24 agencies’ progress in meeting each optimization target, as of February 2017. Following the figure is a more detailed discussion of the progress of each of the 24 agencies.

Among the 24 agencies, SSA and EPA reported the most progress by meeting three targets, 20 reported meeting one or none of the targets, and the remaining 2 agencies did not have a basis to report on progress because they did not have any agency-owned data centers. Of the 22 agencies reporting progress information, 9 were not able to report progress against either the server utilization metric or power usage effectiveness metric, or both, because they lacked the required monitoring tools to measure progress in these areas. OMB began requiring the implementation of these monitoring tools in August 2016; however, as of February 2017, these 9 agencies were not yet reporting implementation of the tools at any of their data centers. This issue is discussed in greater detailed later in this report. Table 2 lists the agencies that met or did not meet each OMB target.

Agencies’ limited progress against OMB’s optimization targets is due, in part, to them not fully addressing our prior recommendations in this area. As noted earlier, in March 2016, we reported on weaknesses in agencies’ data center optimization efforts, including that 22 agencies did not meet OMB’s fiscal year 2015 optimization targets. We noted that this was partially due to the agencies facing challenges in optimizing their data centers, including their decentralized organizational structures that made consolidation and optimization difficult and competing priorities for resources. In addition, consolidating certain data centers was problematic because the volume or type of information involved required the data center to be close in proximity to the users. Accordingly, we recommended that the agencies take action to improve optimization progress, to include addressing any identified challenges. Most agencies agreed with our recommendations or had no comments.

In response to our recommendation, 19 of the 22 agencies submitted corrective action plans to us that described steps they intended to take to improve their data center optimization efforts. Among these steps were developing internal scorecards to track and report on optimization progress, including progress at their component agencies, and launching more aggressive efforts to optimize data centers using virtualization and cloud computing solutions. While 2 of the 22 agencies—Education and HUD—are no longer subject to OMB’s optimization metrics based on OMB’s August 2016 memorandum and their current data center inventory, none of the remaining 20 agencies had fully addressed our recommendation as of May 2017.

The importance of overcoming optimization challenges and addressing our prior recommendations is critical to the ability of agencies to implement the data center optimization provisions of FITARA and achieve OMB’s fiscal year 2018 optimization targets. Going forward, it will be important for the 19 agencies that have established corrective action plans to continue to execute them and monitor the impact of actions completed on their optimization progress. Until agencies fully implement our prior recommendations to address their challenges and improve optimization progress, they may be hindered in implementing the data optimization provisions of FITARA and OMB guidance intended to increase operational efficiency and achieve cost savings. Further, OMB may be challenged in demonstrating that DCOI is meeting its established objectives.

Most Agencies Are Not Planning to Meet OMB’s Fiscal Year 2018 Optimization Targets

In addition to reporting current optimization progress on the Dashboard, OMB requires agencies’ DCOI strategic plans to include, among other things, planned performance levels for fiscal years 2017 and 2018 for each optimization metric.

However, according to the 24 agencies’ DCOI strategic plan information as of April 2017, most are not planning to meet OMB’s optimization targets by the end of fiscal year 2018. More specifically, of the 24 agencies, 5—the Department of Commerce (Commerce), EPA, NSF, the Small Business Administration (SBA), and the U.S. Agency for International Development (USAID)—reported plans to fully meet their applicable targets by the end of fiscal year 2018; 13 reported plans to meet some, but not all, of the targets; 4 reported that they do not plan to meet any targets; and 2 do not have a basis to report planned optimization milestones because they do not report having any agency- owned data centers. Figure 4 summarizes agencies’ progress in meeting OMB’s optimization targets as of February 2017, and planned progress to be achieved by September 2017 and September 2018, as of April 2017.

Agencies’ reported plans to meet the optimization targets also vary by metric. Specifically, about half of the 22 agencies reported plans to meet the facility utilization and virtualization metrics by the end of fiscal year 2018, while less than half are planning to meet the server utilization and automated monitoring, energy metering, and power usage effectiveness metrics. Further, agencies reported that they plan to make the least amount of progress in meeting the target for power usage effectiveness. Figure 5 provides a summary, by optimization metric, of agencies’ current progress in meeting the targets as of February 2017, and planned progress to be achieved by September 2017 and September 2018, as of April 2017.

The limited progress made by agencies in optimizing their data centers, combined with the lack of established plans to improve progress, makes it unclear whether agencies will be able to achieve OMB’s optimization targets by the end of fiscal year 2018. Considering that OMB is expecting at least $2.7 billion in cost savings from agencies’ optimization efforts, the ability of agencies to meet the optimization targets will be critical to meeting this savings goal. However, with less than 2 years remaining until OMB’s fiscal year 2018 DCOI optimization target deadline and the expiration of the data center consolidation and optimization provisions of FITARA in October 2018, only five agencies are planning to meet all of their applicable targets. With the majority of agencies not planning to meet the optimization targets, there is an increased likelihood that agencies will need more time beyond 2018 to continue to implement their optimization efforts. Extending the data center consolidation and optimization provision of FITARA beyond the current October 2018 horizon could provide agencies with additional time to realize the benefits of optimization, including cost savings.

Agencies Reported Successes and Challenges in Optimizing Data Centers

The 24 DCOI agencies reported successes in optimizing their data centers—notably, the benefits of key technologies, such as virtualizing systems to improve performance, and increased energy efficiency. However, agencies also reported operational, technical, and financial challenges related to, for example, improving the utilization of their data center facilities, measuring server utilization, and obtaining funding within their agency for optimization efforts. It will be important for agencies to take action to address their identified challenges—as we previously recommended—in order to improve data center optimization progress.

Agencies Identified the Use of Optimization Technologies and Improved Energy Efficiency as the Most Reported Successes

Agencies reported a variety of successes in optimizing data centers. Specifically, the 24 agencies reported a total of 23 areas of success. Eight areas of successes were identified by three or more agencies, with the most reported successes for an area being identified by 17 agencies. The two most reported areas of success—implementing virtualization technologies and migrating IT applications and services to cloud computing solutions—were similar to the top reported success in achieving consolidation cost savings that we identified in 2014 (i.e., focusing on virtualization and cloud services as consolidation solutions). Agencies are also continuing to report successes in other areas that we highlighted in 2014, including improved energy efficiency, standardized technology, and improved data center inventory reporting. Table 3 details the reported areas of success, as well as the number of related agencies. The most common areas of success are further discussed after the table.

Seventeen agencies reported that implementing virtualization technologies (i.e., running multiple, software-based machines with different operating systems on the same physical machine) has proven successful in optimizing their data centers. For example, officials from Commerce’s Office of the CIO stated that the department had made the most notable optimization progress in virtualizing all non-high performance computing servers, including approximately 11,700 operating systems (as of October 2016). Additionally, officials from Labor’s Office of the CIO noted that virtualization had helped the department create a highly efficient, lower cost, common operating environment suitable for hosting mission-critical applications and services. The officials added that the department expects to significantly increase its migration activity and closures in fiscal years 2017 and 2018 by leveraging the portability of this highly virtualized environment.

As another example, officials from GSA’s IT office stated that the agency has achieved success in retiring older physical systems and shifting to newer, virtualized technologies. The officials stated that these actions have contributed to greater flexibility, stability, and redundancy in the agency’s IT capabilities. Further, officials from NRC’s Office of the CIO stated that their agency had virtualized 72 percent of its servers, which allowed the agency to significantly reduce the amount of old, outdated, and energy-inefficient equipment. Thirteen other agencies also stated that implementing virtualization technologies had led to successes in optimizing their data centers.

About Half of the Agencies Reported That Migrating IT Applications and Services to Cloud Computing Solutions Increased Optimization Progress

Thirteen agencies reported that migrating IT applications and services to cloud computing solutions had led to successes in optimizing their data centers. For example, officials from HHS’s Office of the CIO stated that one of the department’s offices had realized substantial value with the use of cloud-provided solutions, including reducing the cost of data center services by approximately 15 percent compared to government and on- premises data centers.

Additionally, officials from NSF’s Office of Information and Resource Management stated that the agency successfully reduced and streamlined its IT footprint through a number of different efforts, such as migration of applications, e-mail, and instant messaging to cloud providers; networking technology standardization; and server and storage consolidation.

As another example, officials from USAID’s Office of the CIO stated that in 2011 the agency transformed and migrated its primary data center to a private infrastructure cloud provider, thereby eliminating physical infrastructure issues (e.g., power, heating, ventilation, air conditioning, and physical security issues). The officials added that the cloud solution provided the data center infrastructure, network access, connectivity, and other services needed to ensure the delivery of critical business services.

Further, officials from Interior’s Office of the CIO stated that the department had migrated 70,000 users off of 14 legacy e-mail systems to a single department-wide cloud-based e-mail communications and collaboration system. Nine other agencies also stated that migrating to cloud computing solutions led to successes in optimizing their data centers. Their reported successes ranged from migrating e-mail applications to the cloud solutions to responding more timely to shifts in user demand.

Five Agencies Reported Success in Increasing Energy Efficiency to Optimize Their Data Centers

Five agencies reported that increasing their energy efficiency had led to success in optimizing their data centers. For example, officials from the Department of State’s (State) Bureau of Information Resource Management noted that the department has had success in deploying modular data centers that utilize energy-efficient power systems and other optimized operating features that help to reduce the department’s carbon footprint. Further, a program manager from the SSA’s Office of Hardware Engineering stated that the agency had improved its energy efficiency and reduced its carbon footprint through various initiatives including, among other things, rainwater reclamation, improved monitoring of IT equipment power usage, energy-efficient lighting, and the use of solar panels. Figure 6 shows the use of solar panels at the SSA’s National Support Center.

Additionally, officials from EPA’s Office of Environmental Information stated that the agency had success in improving energy efficiency through the purchase of energy efficient IT equipment and by including energy metering in data center facilities planning and buildout to assist with validating energy optimization metrics. The officials also noted that the agency had increased the operating temperature in some data centers as well as used alternate methods of cooling (e.g., outdoor air to cool its data centers), which helped the agency improve its energy efficiency. Officials from Commerce and HHS further stated that increasing their energy efficiency by, for example, purchasing energy- efficient equipment and deploying power monitoring equipment, had led to successes in optimizing their data centers.

Agencies Reported Operational, Technical, and Financial Challenges in Optimizing Their Data Centers

Agencies also reported facing a variety of challenges in optimizing their data centers. Specifically, the 24 DCOI agencies identified a total of 27 types of challenges across three areas: operational, technical, and financial. The highest number of challenges were reported in the operational and technical areas, which included improving data center facility utilization and measuring and reporting on server utilization.

Certain challenges reported were similar to those described to us by agencies in 2016, including those related to competing priorities for labor resources and closing data centers that provide mission critical applications that require proximity to users. Agencies also continued to report operational, technical, and financial challenges that were similar to those described to us in 2014, including gathering data from component agencies, determining power usage information, and obtaining funding from within their agency. For example, in 2014, six agencies noted that gathering data from component agencies was an operational challenge to achieving consolidation cost savings; however, only two agencies are now reporting that as a challenge in optimizing their data centers.

Agencies also cited many new challenges that are specific to optimizing their data centers, such as incorporating enterprise-wide efficiencies when data centers are owned and managed by multiple organizations and the significant upfront costs required to purchase data center monitoring tools. Table 4 details the reported challenges in optimizing data centers, as well as the number of related agencies. The most common challenges are further discussed after the table.

Agencies reported the most operational challenges in the following areas: improving data center facility utilization; competing priorities for labor resources with other agency IT efforts; shifting definitions of a data center and changes to data center optimization requirements; and incorporating enterprise-wide efficiencies when data centers are owned and managed by multiple organizations.

Improving data center facility utilization: Nine agencies cited this challenge. For example, officials from the Department of Veterans Affairs’ (VA) Infrastructure Operations stated that increasing virtualization generally reduces the number of active server racks in the space and, therefore, decreases facility utilization. The officials added that, for smaller rooms that are part of a larger, agency-owned, multi-functional facility, reducing the size of the room is most often not an economical decision, as it does not lead to energy savings or reduced facility costs but, instead, moves the recurring cost of the space from IT to other functions. Officials from DHS, Interior, Labor, and NRC also reported that their increased use of virtualization has negatively impacted their ability to increase facility utilization.

As another example of a challenge in improving facility utilization, officials from Commerce’s Office of the CIO stated that the National Oceanic and Atmospheric Administration’s weather field office data centers contain systems that are proprietary and connect to local weather sensing instruments or satellite communication equipment. The officials said that most of these data centers are averaging only 50 percent facility utilization and have no plans to increase, but are difficult to close because they contain systems designed specifically for the agency’s mission. Officials from Agriculture, GSA, NASA, and the Department of the Treasury (Treasury) also cited challenges in improving facility utilization.

Competing priorities for labor resources with other agency IT efforts: Six agencies cited this challenge. For example, officials from Commerce’s Office of the CIO stated the Census Bureau is ramping up for a very large program—the 2020 Decennial Census—while also working to optimize its data centers. This has led to challenges in implementing data center infrastructure management tools and replacing old power distribution units with new ones. As another example, officials from EPA’s Office of Environmental Information noted that IT personnel are primarily focused on day-to-day operations and maintenance activities and, therefore, resources normally used to support data center activities are periodically pulled away to address more immediate operational activities (such as cybersecurity initiatives).

In addition, SBA’s CIO stated that the biggest challenge faced by the agency is a lack of labor resources, which has historically been due to a focus on mission priorities instead of data center improvements. Officials from GSA, Labor, and Transportation also stated that completing priorities for labor resources has been a challenge to optimizing their data centers.

Shifting definition of a data center and changes to data center optimization requirements: Five agencies cited this challenge. For example, officials from Interior’s Office of the CIO stated that significant changes outlined in OMB’s August 2016 memorandum and previously issued guidance related to the definitions of a data center and optimization metrics presented challenges in maintaining inventories, measuring progress, and assessing cost savings and avoidances. As another example, officials from Energy’s Office of the CIO stated that Energy’s unique computing environments, which support scientific research, facility and plant operations, power management, and mission-specific computing, makes aligning with OMB’s data center definition difficult.

In addition, officials from DHS’s Office of the CIO stated that OMB’s recent changes to the data center optimization metrics, including the focus on agency-owned data centers, greatly impacted the department’s ability to report on optimization progress. More specifically, the officials stated that OMB’s prior optimization metrics focused on the department’s three core data centers (i.e., primary consolidation points); however, under OMB’s new metrics, the department’s core data centers are no longer applicable to the metrics because they are not agency-owned. The officials added that this negatively impacted the department’s ability to report optimization progress related to power usage effectiveness. Officials from GSA and NRC also cited challenges related to the change in the definition of a data center and data center optimization requirements.

Incorporating enterprise-wide efficiencies for data centers owned and managed by multiple organizations: Five agencies identified this challenge. For example, officials from NASA’s Office of the CIO stated that, historically, the agency’s data centers have been owned and managed by multiple organizations, including contractors, which has made it challenging to incorporate enterprise-enabled efficiencies (i.e., common procurements, implementation of standard hardware, software, and management tools). The officials also mentioned that the extensive use of data centers collocated within multi-use buildings, with shared electrical and mechanical infrastructure, has resulted in the agency not realizing the magnitude of savings that would be attributed to the closure of stand-alone data center facilities.

As another example, officials in Energy’s Office of the CIO stated that the implementation of optimization solutions in data centers that are mission and research specific, or have unique operational and environmental requirements, has presented operational challenges. In addition, officials from Justice’s Office of the CIO stated that implementing enterprise solutions across a large and traditionally federated organization has been challenging. Officials from DHS and Labor also cited challenges with improving optimization at data centers that are owned and managed by multiple organizations.

Technical Challenges

Agencies reported the most technical challenges in the following areas: measuring and reporting on server utilization progress, a lack of electricity metering to determine power usage information, and poor network connectivity and low bandwidth at field locations constraining consolidation and optimization efforts.

Measuring and reporting on server utilization progress: Nine agencies cited this challenge. For example, officials from VA’s Infrastructure Operations cited challenges with the complexity of programming the tools needed to collect the data to measure server utilization. In particular, the officials noted issues in delineating what data should be collected to determine server “busy” and “idle” times (e.g., computer processing unit usage, power consumption, or other data) and what unit of time to associate with the data collection (i.e., seconds, minutes, hours, etc.) in order to be able to report on the server utilization metric. As another example, officials from Justice’s Office of the CIO stated that optimizing the server utilization of department data centers that are consolidation points will be extremely difficult because the environments are going through significant changes as they receive servers from other locations.

In addition, officials from Treasury’s Office of the CIO stated that, while the department’s servers have the ability to measure and monitor processing usage, most data centers do not have the ability to centrally aggregate and report on that data. Officials from Agriculture, the Department of Defense (Defense), Labor, NASA, OPM, and Treasury also cited challenges in measuring and reporting on server utilization progress.

Lack of electricity metering to determine power usage information: Seven agencies identified this challenge. For example, officials from Commerce’s Office of the CIO stated that many of the department’s data centers are small and lack separate power metering. The officials added that, rather than adding power monitoring to each small data center, the department needs to conduct further research to evaluate whether consolidation of these unmetered data centers into a few larger well-maintained data centers is more cost effective. As another example, officials from Labor’s Office of the CIO stated that a vast majority of the department’s data centers are in modified office spaces that also serve other purposes, such as accommodating the storage of legacy IT assets and providing a workspace for IT support personnel, which has made the installation of power metering challenging.

Further, officials from VA’s Infrastructure Operations stated that the department’s individual data centers are largely unique, thus requiring detailed engineering to determine how to retrofit energy metering solutions to provide the data necessary for energy usage optimization, particularly without incurring critical IT system downtime. The officials added that the majority of the department’s data centers are not stand-alone data centers, but rather, are rooms within a medical center facility or other multi-purpose facility that were not constructed to facilitate power metering. VA officials stated that these challenges made measuring power usage effectiveness extremely complicated, time-consuming, and costly. Agriculture, Labor, OPM, and SBA also mentioned challenges related to the lack of electricity metering to determine power usage information.

Poor network connectivity and low bandwidth at field locations constrains consolidation and optimization efforts: Five agencies cited this challenge. For example, officials from Interior’s Office of the CIO stated that numerous remote field offices within the department experience poor network connectivity and low bandwidth to support running remotely-hosted applications. The officials added that the risk of reduced service levels at these remote locations is frequently cited as a constraint on consolidation and a challenge to improving optimization progress. As another example, Transportation’s Office of the CIO noted challenges with consolidating field site servers because the telecommunication bandwidth to the field sites is lacking. Officials from HHS, Labor, and SBA also cited concerns about connectivity performance issues as a challenge to consolidation and optimization of data centers at their field office locations.

Financial Challenges

Agencies reported three financial challenges in the following areas: obtaining the funding within their agency for optimization efforts, the upfront costs required to purchase the monitoring tools needed to measure optimization progress, and determining the resulting cost savings and avoidances.

Obtaining the funding within their agency for optimization efforts: Ten agencies cited this challenge. For example, officials from OPM’s Office of the CIO stated that while the agency’s base budget includes ongoing operations and maintenance funding for the agency’s existing data centers, the availability of financial resources during fiscal years 2017 and 2018 would be one of the most significant challenges to improving data center optimization performance, and satisfying DCOI requirements. Further, officials from Justice’s Office of the CIO stated that financial constraints may limit the funding available for migration of component infrastructure to cloud computing services or the department’s core enterprise facilities, which could delay or prevent optimization.

As another example, officials from Defense’s Office of the CIO stated that resource constraints to support application and system rationalization, re-engineering, and migration, forced many component agencies to focus on physical relocations of systems, which limit data center optimization opportunities and savings. Officials from Commerce, DHS, Energy, HHS, Labor, SBA, and Transportation also cited challenges in obtaining the funding within their agency for optimization efforts.

Significant upfront costs required to purchase the monitoring tools needed to measure optimization progress: Eight agencies cited this challenge. For example, officials from Treasury’s Office of the CIO stated that the department is currently in the process of evaluating how to most effectively meet data center power metering requirements without incurring significant expenditures. The officials stated that several of their larger data centers are in older, multi-use buildings and share a cooling infrastructure with the entire building. The officials added that measuring the energy consumed by the portions of the building dedicated to hosting IT equipment would require meters to be installed within just those spaces dedicated to IT, which is a significant cost that is being evaluated relative to other mission-oriented investments.

As another example, officials from Interior’s Office of the CIO stated that the investment for purchasing data center optimization tools would require a reallocation of funds from the department’s fiscal years 2017 and 2018 budgets and would have an adverse effect on meeting other higher priority requirements, such as cybersecurity requirements. Further, the officials stated that purchasing and deploying energy metering tools in the department’s smaller data centers would result in a negative return on investment. In addition, officials from VA’s Infrastructure Operations stated that the department’s data centers are largely unique and require detailed engineering to determine how to retrofit metering solutions to provide data necessary for energy usage optimization, which has not yet been funded. Officials from Agriculture, Commerce, Defense, GSA, and State also cited significant upfront costs of data center monitoring tools as a challenge.

Determining the resulting cost savings and avoidances from consolidation and optimization efforts: Five agencies identified this challenge. For example, officials from NASA’s Office of the CIO stated that due to their extensive use of data centers collocated within multi- use buildings, with shared electrical and mechanical infrastructure, the agency has not realized the magnitude of savings that would be attributed to the closure of stand-alone data center facilities. The officials added that, in most instances, the closed data center spaces have been locally repurposed for non-IT use.

As another example, officials from Agriculture’s Office of the CIO stated that it can be difficult to determine facility costs and the resulting cost savings and avoidances. The officials noted that data centers located within government owned or leased buildings usually do not pay for electricity, heating and air conditioning expenses, or lease and facility upkeep costs, which can present challenges in calculating any cost savings and avoidances from optimization. Officials from GSA, Interior, and Treasury also cited challenges in determining the resulting cost savings and avoidances from their consolidation and optimization efforts.

Addressing these optimization challenges and others—as we previously recommended in 2016—is increasingly important in light of FITARA’s requirements, which direct agencies to establish a multi-year strategic plan to improve data center optimization progress. Until agencies address these challenges, they could be hindered in the implementation of their data center optimization strategic plans and in making initiative-wide progress against OMB’s optimization targets.

Most Agencies Still Cannot Effectively Measure Server Utilization

As noted earlier, FITARA required OMB to establish data center consolidation and optimization metrics, including a metric specific to measuring server efficiency; it also required agencies to report on progress in meeting the metrics. Pursuant to FITARA, OMB’s August 2016 memorandum required agencies to measure and report on server utilization progress, including the number of agency-owned data centers fully equipped with automated monitoring tools and their server utilization percentages.

To effectively measure progress against this metric, OMB’s memorandum also directed agencies to immediately begin replacing the manual collection and reporting of systems, software, and hardware inventory housed within agency-owned data centers with automated monitoring tools and to complete this effort no later than the end of fiscal year 2018. Agencies are required to report progress in implementing automated monitoring tools and server utilization averages at each data center as part of their quarterly data center inventory reporting to OMB. Finally, standards for internal control emphasize the need for federal agencies to establish plans to help ensure goals and objectives can be met, including compliance with applicable laws and regulations.

As of February 2017, 4 of the 22 agencies reporting agency-owned data centers in their inventory—NASA, NSF, SSA, and USAID—reported that they had implemented automated monitoring tools at all of their data centers. Further, 10 reported that they had implemented automated monitoring tools at between 1 and 57 percent of their centers, and 8 had not yet begun to report the implementation of these tools. In total, the 22 agencies reported that automated tools were implemented at 123 (or about 3 percent) of the 4,528 total agency-owned data centers, while the remaining 4,405 (or about 97 percent) of these data centers were not reported as having these tools implemented. Table 5 provides a listing of the number and related percentage of agency-owned data centers reported by agencies as having automated monitoring tools implemented.

Of the 123 data centers reported as having automated monitoring tools implemented, 59 were identified as tiered data centers and 64 as non- tiered data centers. Figure 7 summarizes the number of agency-owned data centers reported with automated monitoring tools installed, including the number of tiered and non-tiered centers.

The limited implementation of automated monitoring tools resulted in incomplete information on server utilization percentages. As noted earlier, OMB’s IT Dashboard is used to publicly report on agencies’ progress in measuring server utilization. This progress information is obtained from agencies’ quarterly data center inventory submissions, which are required to include detailed data on the server utilization averages of each tiered and non-tiered data center. Based on agencies’ February 2017 data center inventory data, 4 of the 22 agencies reported a server utilization average for all of their monitored tiered and non-tiered data centers, 10 reported server utilization averages at a portion of their centers, and 8 did not report this information. SSA reported the highest server utilization average of 100 percent at its one agency-owned tiered data center, while GSA reported the lowest percentage of 9 percent across its 31 agency-owned tiered and non-tiered centers with automated monitoring tools installed.

According to our analysis of agencies’ inventory data, the average server utilization across all 123 data centers with automated monitoring tools installed was about 28 percent, which is approximately 37 percent below OMB’s fiscal year 2018 goal of 65 percent or higher. Figure 8 shows the agency-reported server utilization averages for the 4 agencies that reported this information at all their data centers and the 10 agencies that reported this information at a portion of their centers, as well as the percentage of their agency-owned data centers with automated monitoring tools installed.

For the 18 agencies that did not report server utilization average information at all their data centers, none fully documented plans to implement the automated monitoring tools required to measure this information at all their agency-owned tiered and non-tiered centers by the end of fiscal year 2018. More specifically, our analysis of agencies’ DCOI strategic plans, FITARA implementation milestones, and other documentation (such as project plans and charters) showed that 6 of the 18 agencies—Agriculture, Energy, EPA, GSA, State, and VA—partially documented plans because they addressed implementing automated monitoring tools for only a portion of their data centers. However, these agencies did not address implementing such tools at all tiered and non- tiered agency-owned data centers, as required by OMB. The remaining 12 agencies did not document plans to implement automated monitoring tools. Table 5 provides an assessment of agencies’ documented plans to implement data center automated monitoring tools.

The 18 agencies provided a variety of reasons regarding why they had not established a plan to implement automated monitoring tools at all agency-owned data centers. For example, officials at six agencies (Defense, DHS, EPA, GSA, Labor, and Justice) stated that they were in the process of establishing a plan to implement automated monitoring tools, but had not yet completed it. As another example, agency officials from State’s Bureau of Information Resource Management and NRC’s Office of the CIO noted that they were still evaluating options for purchasing and deploying these tools. Further, officials from OPM’s Office of the CIO and Transportation’s Office of the CIO stated that they were still determining the extent to which their data centers had automated monitoring tools installed. Lastly, officials from Commerce’s Office of the CIO stated the department had no specific plans to invest in automated monitoring tools.

The lack of detailed plans to implement automated monitoring tools at all agency-owned data centers is also due, in part, to OMB not having established a formal requirement to document such plans. Although OMB’s August 2016 memorandum required agencies to submit a DCOI strategic plan by September 30, 2016, and to update it by April 14, 2017, these plans were not required to include detailed information describing how the agency was planning to meet OMB’s requirement to implement automated monitoring tools at all agency-owned tiered and non-tiered centers.

Recognizing this issue, OMB staff from the Office of the Federal CIO stated that they have been advising agencies to include these more detailed plans and milestones for implementing data center automated monitoring tools as part of their publicly available FITARA implementation milestones. However, OMB has not established a formal requirement in its data center guidance or FITARA implementation guidance provided to agencies. As mentioned previously, our analysis of agency’s FITARA implementation milestones showed that most agencies were not aware of OMB’s request to include this information.

Until OMB requires agencies to include detailed plans to implement automated monitoring tools in their FITARA implementation milestones, agencies may continue to lack a roadmap to meet a key DCOI goal. Further, until agencies complete their plans, they may be challenged in implementing the tools needed to effectively measure server utilization—a data center optimization area highlighted in FITARA, and that we previously reported as being critical to improving the efficiency, performance, and environmental footprint of federal data center activities.

Conclusions

With the August 2016 launch of DCOI, OMB took a considerable step forward in providing guidance for the implementation of the data center consolidation and optimization requirements of FITARA and increasing the oversight of agencies’ efforts to optimize their data centers. OMB’s fiscal year 2018 optimization targets provide clear and transparent goals for agencies’ optimization efforts; however, agencies reported limited progress against those targets. Additionally, although agencies’ DCOI strategic plans provide a mechanism for agencies to report planned fiscal years 2017 and 2018 milestones toward achieving OMB’s optimization targets, most agencies reported that they are not planning to meet OMB’s targets by the end of fiscal year 2018. Considering that OMB established a DCOI-wide savings goal of $2.7 billion, the ability of agencies to meet the optimization targets will be critical to achieving these savings. Extending the time frame for the agencies to meet the required data center consolidation and optimization provisions of FITARA beyond October 2018 could provide agencies with additional time to achieve the benefits of optimization. In addition, agencies’ implementation of our prior recommendations to address optimization challenges and improve progress could help ensure that they are better positioned to meet key DCOI goals.

As a result of OMB’s increased focus on data center optimization beginning in 2013 and its more recent efforts to launch DCOI, agencies have reported noteworthy successes in optimizing their data centers— particularly in leveraging virtualization and cloud computing as a means to optimize their data centers. These constructive experiences indicate that DCOI is moving in the right direction. However, as agencies work toward achieving OMB’s fiscal year 2018 optimization targets, many are reporting challenges related to improving data center facility utilization, measuring and reporting on server utilization progress, and obtaining the funding within their agency for optimization efforts. Such a dynamic environment reinforces the need for agencies to address their identified challenges— as we previously recommended—in order to improve data center optimization progress.

OMB’s efforts to establish a metric to measure server utilization as part of its August 2016 memorandum were consistent with our 2014 recommendation and an important step toward ensuring that agency computing resources are being used more efficiently. Additionally, OMB’s requirement that agencies implement automated monitoring tools at their data centers by the end of fiscal year 2018 will help to ensure that they have the necessary foundation in place to effectively measure and report on server utilization progress. However, with agencies collectively reporting that these tools are only installed at about 3 percent of the total data centers and with 18 agencies lacking complete plans to implement these tools at their remaining data centers, significant work remains toward meeting OMB’s requirement. The lack of a formal OMB requirement to establish detailed plans in this area and report them to OMB further increases the likelihood that agencies will continue to lack them.

In the absence of such a requirement and completed plans, agencies will be missing an important roadmap for implementing the automated monitoring tools needed to measure server utilization—an area that both we and OMB have reported as critical to improving the efficiency, performance, and environmental footprint of federal data center activities. Moreover, with automated monitoring tools not required by OMB to be fully implemented by agencies until the end of fiscal year 2018, extending the time frame of FITARA’s data center consolidation and optimization provisions could also better ensure that server utilization is effectively measured and reported beyond fiscal year 2018, after the necessary monitoring tools are implemented.

Matter for Congressional Consideration

As most agencies lack plans to meet OMB’s data center optimization targets by the end of fiscal year 2018, it is increasingly likely that these agencies will require additional time to achieve the data center consolidation and optimization goals required by FITARA and OMB guidance. In order to provide agencies with additional time to meet OMB’s data center optimization targets and achieve the related cost savings, Congress should consider extending the time frame for the data center consolidation and optimization provisions of FITARA beyond their current expiration date of October 1, 2018.

Recommendations for Executive Action

To better ensure that agencies complete important DCOI planning documentation and that the initiative improves governmental efficiency and achieves intended cost savings, we are recommending that the Director of OMB direct the Federal CIO to formally document a requirement for agencies to include plans, as part of existing OMB reporting mechanisms, to implement automated monitoring tools at their agency-owned data centers.

We are also recommending that the Secretaries of Agriculture, Commerce, Defense, Homeland Security, Energy, HHS, Interior, Labor, State, Transportation, Treasury, and VA; the Attorney General of the United States; the Administrators of EPA, GSA, and SBA; the Director of OPM; and the Chairman of NRC take action to, within existing OMB reporting mechanisms, complete plans describing how the agency will achieve OMB’s requirement to implement automated monitoring tools at all agency-owned data centers by the end of fiscal year 2018.

Agency Comments and Our Evaluation

We received comments on a draft of this report from OMB and the 24 agencies that we reviewed. Of the 19 agencies to which we made recommendations, 10 agencies agreed with our recommendations, 3 (Defense, Interior, and OPM) partially agreed, and 6 (including OMB) did not state whether they agreed or disagreed. In addition, 6 agencies to which we did not make recommendations stated that they had no comments. Multiple agencies also provided technical comments, which we have incorporated as appropriate.

The following discusses the comments from each agency to which we made a recommendation.

In an e-mail received on July 7, 2017, a staff member from OMB’s Office of General Counsel stated that the agency had no comments on the draft report. The staff member did not state whether the agency agreed or disagreed with our recommendation.

In an e-mail received on June 26, 2017, a senior advisor in the Department of Agriculture’s Office of the CIO did not state whether the department agreed or disagreed with our recommendation, but noted that the department understands that automated monitoring of server utilization and virtualization is critical to accurate data center performance and cost savings reporting.

In written comments, Commerce stated that it agreed with our recommendation and described actions planned to implement it. Specifically, the department noted that, as part of its effort to consolidate, define, and establish a plan to deploy an enterprise-wide automated monitoring tool, it has identified two component agencies that will offer a data center infrastructure management tool as a service. The department added that this approach will allow it to monitor and report cost savings and avoidances more efficiently. Commerce’s comments are reprinted in appendix II.

In written comments, Defense stated that it partially agreed with our recommendation. Specifically, the department stated that it recognizes the value of data center infrastructure management capabilities in realizing DCOI objectives and will endeavor to implement the capabilities as quickly as possible. However, the department noted that it will be unable to complete the implementation of data center infrastructure management capabilities by the end of fiscal year 2018, as we recommended. As obstacles to meeting this deadline, the department cited procurement regulations, resource challenges, the budget cycle, and remaining work to resolve the population of installation processing nodes, but did not offer further details.

Our report specifically recognizes the challenges cited by agencies in the implementation of automated monitoring tools (i.e., data center infrastructure management capabilities), and notes the importance of detailed plans to overcome these challenges. Given the department’s own acknowledgment of facing implementation obstacles, a plan describing how it will implement these important monitoring tools could help overcome the challenges identified. Therefore, we continue to believe our recommendation is warranted. Defense’s comments are reprinted in appendix III.

In written comments, Energy stated that the department concurred with our recommendation and described planned actions to implement it. Specifically, the department stated that it established plans to implement automated monitoring tools at its 78 department-owned tiered data centers and plans to evaluate whether its 68 department- owned non-tiered data centers should be consolidated or closed. For the non-tiered centers slated to remain open, the department stated that it expects to complete plans describing how it will automate server utilization by September 2019. Energy’s comments are reprinted in appendix IV.

In written comments, HHS stated that the department concurred with our recommendation and described planned actions to implement it. Specifically, the department stated that HHS will direct its operating and staff divisions to acquire and install automated monitoring tools in all agency-owned data centers by the close of fiscal year 2018. HHS’s comments are reprinted in appendix V.

In written comments, DHS stated that the department concurred with our recommendation and described planned actions to implement it. Specifically, the department stated that it is continually reviewing optimization alternatives, including evaluating the option to move to a cloud deployment model over the next few years. The department further noted that it does not expect to achieve the optimum solution in agency-owned tiered data centers by the end of fiscal year 2018, as we recommended, but agreed with our suggestion that the DCOI time frame be reconsidered. In addition, DHS stated that it expects to have an optimization plan that includes, among other things, resource requirements and a schedule to achieve monitoring compliance for agency-owned tiered data centers by April 2018. DHS’s comments are reprinted in appendix VI.

In written comments, Interior stated that the department partially concurred with our recommendation. Specifically, the department stated that it is committed to completing its plan on schedule, but that its ability to meet OMB’s requirement to implement automated monitoring tools at all department-owned data centers by the end of fiscal year 2018, as we recommended, will depend on many factors and variables, including the availability of funding and other resources.

Because of the potential for improved efficiency and cost savings from data center optimization, as discussed in this report, we believe the department should devote the necessary resources to ensure that automated monitoring tools are installed at all department-owned data centers by the end of fiscal year 2018, as required by OMB. Therefore, in our view, the recommendation continues to be warranted. Interior’s comments are reprinted in appendix VII.

In an e-mail received on July 13, 2017, a Justice audit liaison stated that the department concurred with our recommendation.

In written comments, Labor stated that the department accepted our recommendation and will incorporate pertinent information in its next data center consolidation and optimization strategic plan due in April 2018. Labor’s comments are reprinted in appendix VIII.

In written comments, State indicated that the department agreed with our recommendation and described completed and planned actions to address it. Specifically, the department stated that it performed an analysis of tools, including shared services and commercial-off-the- shelf products. The department also stated that it is developing an acquisition strategy based on its research and is recommending that a commercially available product would be the best solution to meet monitoring requirements. Further, the department noted that additional budgetary resources may be required to support an enterprise-wide roll-out of automated server monitoring across all tiered data centers, which may not be available until fiscal year 2019 or later.

As discussed in detail in this report, data center optimization holds the potential for improved efficiency and cost savings. Consequently, we encourage the department to devote the necessary resources to ensure that automated monitoring tools are installed at all department- owned data centers by the end of fiscal year 2018, as required by OMB. State’s comments are reprinted in appendix IX.

In an e-mail received on July 3, 2017, a deputy director in Transportation’s Audit Relations and Program Improvement office stated that the department concurred with our recommendation.

In an e-mail received on July 20, 2017, an audit liaison in Treasury’s Office of the CIO stated that the department had no comments on the draft report, and did not state whether the agency agreed or disagreed with our recommendation.

In written comments, VA stated that it concurred with our recommendation and noted that it is developing a plan to fully comply with OMB’s requirement to implement automated monitoring tools at all agency-owned data centers by the end of fiscal year 2018. The department added that it expects to complete this plan by November 2017. VA’s comments are reprinted in appendix X.

In written comments, EPA did not state whether the agency agreed or disagreed with our recommendation, but described planned actions to implement it. Specifically, the agency detailed plans to address OMB's requirements, such as leveraging EPA's current investment in a network monitoring tool and the intent to procure and deploy a data center infrastructure management tool by the end of fiscal year 2018. However, EPA also noted that budget cuts may delay the agency's efforts to fully implement the requirements of DCOI.

As noted earlier, because of the potential efficiency and savings from data center optimization, we believe EPA should devote the necessary resources to ensure that automated monitoring tools are installed at all department-owned data centers by the end of fiscal year 2018, as required by OMB. EPA's written comments are reprinted in appendix XI.

In written comments, GSA stated that it agreed with our recommendation and that it plans to install automated monitoring tools by the end of fiscal year 2018. GSA’s comments are reprinted in appendix XII.

In written comments, NRC stated that it was in general agreement with our findings. The agency did not state whether it agreed or disagreed with our recommendation, but described actions planned to address it. Specifically, the agency stated that it plans to install automated monitoring tools in all of its tiered data centers. The agency added that it is planning to close its non-tiered data centers. NRC’s comments are reprinted in appendix XIII.

In written comments, OPM stated that the agency partially concurred with our recommendation. Specifically, the agency stated that it plans to consolidate its remaining data centers into two main locations by the end of fiscal year 2018. OPM further stated that this consolidation will obviate the need to implement automated monitoring tools at the data centers that are closing. Finally, the agency noted that it is implementing automated monitoring tools at the designated core data centers.

We encourage OPM’s efforts to continue to consolidate its data centers. However, as mentioned in its comments, OPM’s automated monitoring tools have not yet been installed at the agency’s core data centers. Completing a plan describing how the agency will meet OMB’s requirement to implement automated monitoring tools at these centers, as we recommended, could better ensure that this important effort is completed. Therefore, we believe our recommendation is still warranted. OPM’s comments are reprinted in appendix XIV.

In an e-mail received on July 13, 2017, a program manager in SBA’s Office of Congressional and Legislative Affairs stated that the agency had no comments on the draft report, and did not state whether the agency agreed or disagreed with our recommendation.

In addition to the aforementioned comments, six agencies to which we did not make recommendations provided the following responses: In an e-mail received on June 23, 2017, a policy analyst in Education’s Office of the Secretary/Executive Secretariat stated that the department had no comments on the draft report.

In written comments, HUD stated that the department had no comments on the draft report. HUD’s comments are reprinted in appendix XV.

In an e-mail received on July 14, 2017, a NASA audit liaison stated that the agency had no comments on the draft report.

In an e-mail received on July 17, 2017, a NSF audit liaison stated that the agency had no comments on the draft report.

In written comments, SSA stated that the agency had no comments on the draft report. SSA’s comments are reprinted in appendix XVI.

In an e-mail received on July 12, 2017, an audit liaison in USAID’s Bureau for Management stated that the agency had no comments on the draft report.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 22 days from the report date. At that time, we will send copies to interested congressional committees, the Director of OMB, Secretaries and agency heads of the departments and agencies addressed in this report, and other interested parties. In addition, the report will be available at no charge on GAO’s website at http://www.gao.gov.

If you or your staffs have any questions on the matters discussed in this report, please contact me at (202) 512-9286 or pownerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix XVII.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to (1) assess agencies’ progress against the Office of Management and Budget’s (OMB) data center optimization targets, (2) identify agencies’ notable optimization successes and challenges, and (3) evaluate the extent to which agencies are able to effectively measure server utilization.

To assess agencies’ progress against OMB’s data center optimization targets, we analyzed the February 2017 data center optimization progress information of the 24 department and agencies (agencies) that participate in OMB’s Data Center Optimization Initiative (DCOI). This progress information was obtained from the Information Technology (IT) Dashboard—an OMB public website that provides information on federal agencies’ major IT investments. We then compared the agencies’ optimization progress information against OMB’s fiscal year 2018 optimization targets, as documented in its August 2016 memorandum. Although OMB’s memorandum establishes a single optimization target value for the server utilization and automated monitoring metric, the Dashboard displays agencies’ progress for tiered and non-tiered data centers separately. To report consistently with OMB’s implementation memorandum, we combined the progress information for tiered and non- tiered data centers into a single assessment in this report.

We also reviewed the 24 agencies’ DCOI strategic plans, as of April 2017, to obtain information regarding their fiscal years 2017 and 2018 plans to meet or not meet OMB’s optimization targets. This documentation included agencies’ strategic plan information publicly posted on agency-owned digital strategy websites, and additional agency- provided documentation of their data center consolidation and optimization strategic plans.

To assess the reliability of agencies’ optimization progress information on OMB’s IT Dashboard, we reviewed the information for errors or missing data, such as progress information that was not available for certain metrics. We also compared agencies’ optimization progress information across multiple reporting quarters to identify any inconsistencies in agencies progress. We discussed with OMB staff any discrepancies or potential errors identified to determine the causes or request additional information. In addition, we interviewed OMB officials to obtain additional information regarding the steps taken to ensure the reliability of and validate the optimization data on the Dashboard. We determined that the data were sufficiently reliable to report on agencies’ optimization progress.

To assess the reliability of the DCOI strategic plans, we reviewed agencies’ documentation to identify any missing data or errors. We also compared the planned data center optimization milestones in agencies’ documentation against current optimization progress information obtained from the Dashboard. In addition, we reviewed agency chief information officer statements attesting to the completeness of their DCOI strategic plan information. Moreover, we obtained written responses from agency officials regarding the steps taken to ensure the accuracy and reliability of their strategic plan. We discussed with agency officials any discrepancies or potential errors identified during our reviews of their strategic plan to determine the causes or request additional information. As a result of these efforts, we determined that the agencies’ strategic plan information was sufficiently reliable for reporting on plans to meet or not meet OMB’s fiscal year 2018 optimization targets.

To address the second objective, we reviewed the 24 agencies DCOI strategic plans to identify successes and challenges encountered by agencies in optimizing their data centers. We also interviewed cognizant officials at the 24 agencies in order to gather additional information about their data center optimization successes and challenges. We then categorized the agency-reported successes and challenges to determine the ones encountered most often.

To evaluate the extent to which selected agencies are able to effectively measure server utilization, we analyzed the 24 agencies’ February 2017 data center inventory information. We reviewed the inventory information to determine the extent to which the agencies reported the implementation of automated monitoring tools at their data centers to measure server utilization, as well as the reported server utilization percentages at those centers. To determine whether agencies had established detailed plans to meet OMB’s M-16-19 requirement to implement automated monitoring tools at all agency-owned data centers by the end of fiscal year 2018, we reviewed agencies DCOI strategic plans, publicly available milestone information for implementing the December 2014 IT acquisition reform law, and other planning documentation provided by agencies (such as project charters and project plans). We reviewed this documentation to determine the extent to which agencies documented plans to implement automated monitoring tools at all their agency-owned data centers by the end of fiscal year 2018, as required by OMB.

To assess the reliability of the agencies’ data center inventories, we checked for missing data and other errors, such as anomalous server utilization percentage information. We also compared agencies’ reported use of automated monitoring tools at their data centers across multiple reporting quarters to identify any inconsistencies in agencies’ progress. We discussed with agency officials any discrepancies or potential errors identified to determine the causes or request additional information. Further, we obtained written responses from agency officials regarding actions taken to ensure the reliability of their inventory data. We determined that the agencies’ data were sufficiently reliable to report on agencies’ progress in implementing automated monitoring tools to measure server utilization.

We conducted this performance audit from July 2016 to August 2017 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Department of Commerce

Appendix III: Comments from the Department of Defense

Appendix IV: Comments from the Department of Energy

Appendix V: Comments from the Department of Health and Human Services

Appendix VI: Comments from the Department of Homeland Security

Appendix VII: Comments from the Department of the Interior

Appendix VIII: Comments from the Department of Labor

Appendix IX: Comments from the Department of State

Appendix X: Comments from the Department of Veterans Affairs

Appendix XI: Comments from the Environmental Protection Agency

Appendix XII: Comments from the General Services Administration

Appendix XIII: Comments from the Nuclear Regulatory Commission

Appendix XIV: Comments from the Office of Personnel Management

Appendix XV: Comments from the Department of Housing and Urban Development

Appendix XVI: Comments from the Social Security Administration

Appendix XVII: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, individuals making contributions to this report included Dave Hinchman (Assistant Director), Jon Ticehurst (Assistant Director), Chris Businsky, Rebecca Eyler, Linda Kochersberger, and Jonathan Wall.