Data Center Optimization: Agencies Need to Address Challenges and Improve Progress to Achieve Cost Savings Goal
Published: 2017-08-15T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: In December 2014, FITARA was enacted and included a series of provisions related to improving the performance of data centers, including requiring OMB to establish optimization metrics and agencies to report on progress toward meeting the metrics. OMB's Federal Chief Information Officer subsequently launched DCOI to build on prior data center consolidation and optimization efforts.
GAO was asked to review data center optimization. GAO's objectives were to (1) assess agencies' progress against OMB's data center optimization targets, (2) identify agencies' notable optimization successes and challenges, and (3) evaluate the extent to which agencies are able to effectively measure server utilization. To do so, GAO evaluated the 24 DCOI agencies' progress against OMB's fiscal year 2018 optimization targets, interviewed officials, and assessed agencies' efforts to implement monitoring tools for server utilization.
section_title: What GAO Found
paragraphs: Of the 24 agencies required to participate in the Office of Management and Budget's (OMB) Data Center Optimization Initiative (DCOI), 22 collectively reported limited progress against OMB's fiscal year 2018 performance targets. Two agencies did not have a basis to report on progress as they do not have agency-owned data centers. For OMB's five optimization targets, five agencies or less reported that they met or exceeded each of the targets (see figure).
Further, as of April 2017, 17 of the 22 agencies were not planning to meet OMB's targets by September 30, 2018. This is concerning because the Federal Information Technology Acquisition Reform Act's (FITARA) data center consolidation and optimization provisions, such as those that require agencies to report on optimization progress and cost savings, expire a day later on October 1, 2018. Extending the time frame of these provisions would increase the likelihood that agencies will meet OMB's optimization targets and realize related cost savings. Additionally, until agencies improve their optimization progress, OMB's $2.7 billion initiative-wide cost savings goal may not be achievable.
All 24 agencies reported successes in optimizing their data centers—notably, the benefits of key technologies, such as virtualizing systems to improve performance, and increased energy efficiency. However, agencies also reported challenges related to, for example, improving the utilization of their data center facilities and competing for labor resources. It will be important for agencies to take action to address their identified challenges—as GAO previously recommended—in order to improve data center optimization progress.
Of the 24 agencies required by OMB to implement automated monitoring tools to measure server utilization by the end of fiscal year 2018, 4 reported in their data center inventories as of February 2017 that they had fully implemented such tools, 18 reported that they had not, and 2 did not have a basis to report on progress because they do not have agency-owned data centers. Collectively, agencies reported that these tools were used at about 3 percent of their centers. Although federal standards emphasize the need to establish plans to help ensure goals are met, of the 18 agencies, none fully documented plans, 6 agencies had partially documented them, and 12 did not document them. Agencies provided varied reasons for this, including that they were still evaluating available tools. In addition, the lack of a formal requirement from OMB to establish the plans also contributed to agencies not having them. Until these plans are completed, agencies may be challenged in measuring server utilization.
section_title: What GAO Recommends
paragraphs: Congress should consider extending the time frame for the data center consolidation and optimization provisions of FITARA to provide agencies with additional time to meet OMB's targets and achieve cost savings. GAO is also recommending that 18 agencies complete their plans to implement data center monitoring tools and that OMB require agencies to complete their plans and report them to OMB. Ten agencies agreed with GAO's recommendations, three agencies partially agreed, and six (including OMB) did not state whether they agreed or disagreed, as discussed in the report.

section_title: Letter
paragraphs: The federal government’s demand for information technology (IT) is ever  increasing. In recent years, as federal agencies have modernized their  operations, put more of their services online, and improved their  information security profiles, their need for computing power and data  storage resources has grown. Accordingly, this growing demand has led  to a dramatic rise in the number of federal data centers and a  corresponding increase in operational costs. In response, the Office of  Management and Budget’s (OMB) Federal Chief Information Officer (CIO)  launched the Federal Data Center Consolidation Initiative (FDCCI) in  2010 to reduce the growing number of centers.
Congress has also recognized the importance of reforming the  government-wide management of IT, and in December 2014, Federal  Information Technology Acquisition Reform provisions (commonly  referred to as FITARA) were enacted as a part of the Carl Levin and  Howard P. ‘Buck’ McKeon National Defense Authorization Act for Fiscal  Year 2015. The law includes specific requirements related to federal  data center optimization including, for example, that OMB is to establish  metrics to measure data center optimization progress (to include server  efficiency).
Pursuant to FITARA, in August 2016, the Federal CIO issued a  memorandum that announced the Data Center Optimization Initiative  (DCOI) as a successor effort to FDCCI. According to OMB, this new  initiative supersedes and builds on the results of FDCCI, and is also  intended to improve the performance of federal data centers in areas  such as facility utilization and power usage. Among other things, DCOI  requires 24 federal departments and agencies (agencies) to develop  plans and report on strategies (referred to as DCOI strategic plans) to  consolidate inefficient infrastructure, optimize existing facilities, improve  security posture, and achieve cost savings.
Over the past several years, we have reported and testified that, while  data center consolidation and optimization could potentially save the  federal government billions of dollars, weaknesses exist in the execution  and oversight of these efforts. For example, in March 2016, we reported  that 22 agencies had collectively made limited progress against OMB’s  fiscal year 2015 data center optimization performance metrics. As a  result, we recommended that these agencies take action to improve  optimization progress. Most agencies agreed with our recommendations  or had no comments.
Given the importance of the optimization initiative, this report responds to  the committees’ request that we review federal agencies’ data center  optimization progress. The specific objectives of this review were to (1)  assess agencies’ progress against OMB’s data center optimization  targets, (2) identify agencies’ notable optimization successes and  challenges, and (3) evaluate the extent to which agencies are able to  effectively measure server utilization.
To address the first objective, we analyzed 24 DCOI agencies’ February  2017 data center optimization progress information from the IT  Dashboard—an OMB public website that provides information on federal  agencies’ major IT investments. We then compared the agencies’  optimization progress information against OMB’s fiscal year 2018  optimization targets, as documented in its August 2016 memorandum.  We also reviewed the 24 agencies’ DCOI strategic plans, as of April  2017, to obtain information regarding their fiscal years 2017 and 2018  plans to meet or not meet OMB’s optimization targets.
To address the second objective, we reviewed the 24 agencies’ DCOI  strategic plans to identify successes and challenges encountered by  agencies in optimizing their data centers. We also interviewed cognizant  officials at the 24 agencies in order to gather additional information about  their data center optimization successes and challenges. We then  categorized the agency-reported successes and challenges to determine  the ones encountered most often.
For the third objective, we analyzed the 24 agencies’ February 2017 data  center inventory information to determine the extent to which the  agencies reported the implementation of automated monitoring tools at  their data centers to measure server utilization, as well as the reported  server utilization percentages at those centers. We also reviewed  agencies’ DCOI strategic plans, FITARA implementation milestone  information, and other planning documentation provided by agencies  (such as project charters and project plans) to determine the extent to  which agencies documented plans to implement automated monitoring  tools at all their data centers by the end of fiscal year 2018, as required  by OMB’s August 2016 memorandum. See appendix I for a more detailed  discussion of our objectives, scope, and methodology.
We conducted this performance audit from July 2016 to August 2017 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: The federal government’s increasing demand for IT has led to an  increase in the number of federal data centers and a corresponding  increase in operational costs. According to OMB, the federal government  reported 432 data centers in 1998, 2,094 in July 2010, and 9,995 in  August 2016. Operating such a large number of centers has been and  continues to be a significant cost to the federal government, including  costs for hardware, software, real estate, and cooling. For example, in  2007, the Environmental Protection Agency (EPA) estimated that the  electricity costs to operate federal servers and data centers across the  government were about $450 million annually. According to the  Department of Energy (Energy), a typical data center has 100 to 200  times the energy use intensity of a commercial building. In 2009, OMB  reported that server utilization rates as low as 5 percent across the  federal government’s estimated 150,000 servers were a factor driving the  need to establish a coordinated, government-wide effort to improve the  efficiency, performance, and environmental footprint of federal data  center activities.
subsections: section_title: OMB and the Federal CIO Established FDCCI
paragraphs: Concerned about the size of the federal data center inventory and the  potential to improve the efficiency, performance, and the environmental  footprint of federal data center activities, OMB, under the direction of the  Federal CIO, established FDCCI in February 2010. This initiative’s four  high-level goals were to promote the use of “green IT” by reducing the overall energy and real  estate footprint of government data centers;  reduce the cost of data center hardware, software, and operations;  increase the overall IT security posture of the government; and  shift IT investments to more efficient computing platforms and  technologies.
As part of the initiative, OMB required the 24 agencies to identify a data  center consolidation program manager to lead the agency’s consolidation  efforts. In addition, agencies were required to submit an asset inventory  baseline and other documents that would result in a plan for consolidating  their data centers. The asset inventory baseline was to contain detailed  information on each data center and identify the consolidation approach  to be taken for each one. It would serve as the foundation for developing  the final data center consolidation plan. The data center consolidation  plan would serve as a technical road map and approach for achieving the  targets for infrastructure utilization, energy efficiency, and cost efficiency.
In October 2010, OMB reported that all of the agencies had submitted an  inventory and plan. OMB also clarified the definition of a data center and  noted that, for the purposes of FDCCI, a data center is defined as any  room used for the purpose of processing or storing data that is larger than  500 square feet and meets stringent availability requirements. Under this  definition, OMB reported that agencies had identified 2,094 data centers  as of July 2010. “…a data center is…a closet, room, floor, or building for the storage, management, and  dissemination of data and information and  computer systems and  associated components, such as database, application, and storage systems and data  stores [excluding facilities exclusively devoted to communications and network equipment  (e.g., telephone exchanges and telecommunications rooms)]. A data center generally  includes redundant or backup power supplies, redundant data communications  connections, environmental controls…and special security devices housed in leased,  owned, collocated, or stand-alone facilities.”
Under the new definition, OMB estimated that there were a total of 3,133  federal data centers in December 2011, and its goal was to consolidate  approximately 40 percent, or 1,253 data centers, for a savings of  approximately $3 billion by the end of 2015. See figure 1 for an example  of an image of data center server racks at the Social Security  Administration’s (SSA) National Support Center.
In March 2012, OMB launched the PortfolioStat initiative, which requires  agencies to conduct an annual agency-wide IT portfolio review to, among  other things, reduce commodity IT spending and demonstrate how its IT  investments align with the agency’s mission and business functions.  PortfolioStat is designed to assist agencies in assessing the current  maturity of their IT portfolio management process, make decisions on  eliminating duplication, and move to shared solutions in order to  maximize the return on IT investments across the portfolio.
Subsequently, in March 2013, OMB issued a memorandum that  documented the integration of FDCCI with PortfolioStat and stated that  agencies should focus on an enterprise-wide approach to address  commodity IT (including data centers) in a comprehensive manner. The  memorandum also discussed consolidating previously collected IT-related  plans, reports, and data submissions. For example, agencies were no  longer required to submit the data center consolidation plans previously  required in 2012. However, OMB required agencies to update their data  center inventories and report on consolidation progress at the end of  every quarter.
OMB’s 2013 memorandum also increased the focus on optimizing the  performance of federal data centers. Specifically, OMB stated that, to  more effectively measure the efficiency of an agency’s data center  assets, agencies would also be measured by the extent to which their  primary data centers were optimized for total cost of ownership by  incorporating metrics for data center energy, facility, labor, and storage,  among other things. Subsequently, in May 2014, OMB issued  memorandum M-14-08, which established a set of data center  optimization metrics to measure agency progress. In addition, OMB  established target values that agencies were expected to achieve by the  end of fiscal year 2015.
section_title: IT Acquisition Reform Law Enhanced Data Center Consolidation and Optimization Efforts
paragraphs: Recognizing the importance of reforming the government-wide  management of IT, Congress enacted FITARA in December 2014.  Among other things, the law includes a number of requirements related to  federal data center consolidation and optimization:
Agencies shall submit to OMB a comprehensive inventory of the data  centers owned, operated, or maintained by or on behalf of the agency.
Agencies shall submit a multi-year strategy to achieve the  consolidation and optimization of the agency’s data centers no later  than the end of fiscal year 2016. This strategy should include, for  example, performance metrics that are consistent with the  government-wide data center consolidation and optimization metrics.
On a quarterly basis, agencies shall report to OMB’s Administrator of  the Office of Electronic Government on progress towards meeting  government-wide data center consolidation and optimization metrics.
OMB’s Administrator of the Office of Electronic Government shall  establish metrics applicable to the consolidation and optimization of  data centers (including server efficiency), ensure that agencies’  progress toward meeting government-wide data center consolidation  and optimization metrics is made publicly available, review agencies’  inventories and strategies to determine whether they are  comprehensive and complete, and monitor the implementation of  each agency’s strategy.
Not later than December 19, 2015, OMB’s Administrator of the Office  of Electronic Government shall develop and make publicly available, a  goal, broken down by year, for the amount of planned cost savings  and optimization improvements achieved through FDCCI and, for  each year thereafter through October 1, 2018, compare reported cost  savings and optimization improvements against those goals.
The law’s data center consolidation and optimization provisions expire on  October 1, 2018.
In June 2015, OMB memorandum M-15-14 provided guidance for  implementing FITARA and related IT management practices. OMB’s  guidance includes several actions that agencies are to take to establish a  basic set of roles and responsibilities (referred to as the “common  baseline”) for CIOs and other senior agency officials that are needed to  implement the authorities described in the law. For example, agencies are  to conduct a self-assessment to identify where they conform to the  common baseline and where they deviate. OMB guidance also requires  agencies to annually update their self-assessments and report their  progress in reaching FITARA implementation milestones.
section_title: OMB Established DCOI
paragraphs: In August 2016, OMB issued a memorandum that established DCOI  and included guidance on how to implement the data center consolidation  and optimization provisions of FITARA. Among other things, the guidance  requires agencies to consolidate inefficient infrastructure, optimize  existing facilities, improve their security posture, and achieve cost  savings. For example, agencies are required to maintain a complete  inventory of all data center facilities owned, operated, or maintained by or  on behalf of the agencies and measure progress toward defined  optimization performance metrics on a quarterly basis as part of their data  center inventory submissions.
OMB’s August 2016 memorandum also revised the definition of a  physical data center to include any room with at least one server that  provides services (such as testing and development). Further, OMB’s  guidance directed agencies to categorize their data centers as either a  tiered data center or a non-tiered data center. OMB guidance defines a  tiered data center as one that uses each of the following: a separate  physical space for IT infrastructure, an uninterruptible power supply, a  dedicated cooling system or zone, and a backup power generator for a  prolonged power outage. According to OMB, all other data centers shall  be considered non-tiered.
Regarding data center optimization planning, the memorandum directs  agencies to develop DCOI strategic plans that define their data center  strategies for fiscal years 2016 through 2018. Among other things, this  strategy is to include a timeline for agency consolidation and optimization  activities with an emphasis on cost savings and optimization performance  benchmarks the agency can achieve between fiscal years 2016 and  2018. For example, agencies are required to establish planned data  center optimization milestones and report on progress toward achieving  those milestones in their strategic plans. OMB required agencies to  publicly post the plans to their agency-owned digital strategy websites by  September 30, 2016, and to post subsequent strategic plan updates by  April 14, 2017, and April 13, 2018.
OMB also directed agencies to update their publicly available FITARA  implementation milestone information to identify, at a minimum, five  milestones per fiscal year to be achieved through DCOI. According to  OMB, the DCOI milestones are expected to be updated quarterly as  progress is achieved and are to be reviewed in quarterly meetings with  OMB staff.
Further, the memorandum states that OMB will report government-wide  and agency-specific progress on the IT Dashboard—a public website that  provides detailed information on major IT investments. According to OMB,  this progress information is to include planned and achieved data center  closures, consolidation-related costs savings, and data center  optimization performance information. In this regard, OMB began  including data center consolidation and optimization progress information  on the Dashboard in August 2016.
Moreover, OMB guidance includes a series of performance metrics in the  areas of data center closures, cost savings, and optimization progress.
Data center closures: Agencies are expected to close at least 25  percent of tiered data centers government-wide, excluding those  approved as inter-agency shared services providers, by the end of  fiscal year 2018. Further, agencies are to close at least 60 percent of  non-tiered data centers government-wide by the end of fiscal year  2018. OMB’s guidance further notes that, in the long term, all  agencies should continually strive to close all non-tiered data centers,  noting that server rooms and closets pose security risks and  management challenges and are an inefficient use of resources.
Cost savings: Agencies are expected to reduce government-wide  annual costs attributable to physical data centers by at least 25  percent, resulting in savings of at least $2.7 billion, by the end of fiscal  year 2018.
Data center optimization: Agencies are expected to measure progress  against a series of new data center performance metrics in the  areas of server utilization, energy metering, power usage, facility  utilization, and virtualization. Further, OMB’s guidance establishes  target values for each metric that agencies are to achieve by fiscal  year 2018.
To improve the measurement of data center optimization progress,  OMB’s memorandum directs agencies to replace the manual collection  and reporting of systems, software, and hardware inventory housed within  data centers with automated monitoring, inventory, and management  tools (e.g., data center infrastructure management) by the end of fiscal  year 2018. According to OMB, these data center tools (henceforth  referred to as “automated monitoring tools”) are to provide the capability  to, at a minimum, measure progress toward server utilization and  virtualization metrics. While implementation of automated monitoring tools  is not required to be completed until the end of fiscal year 2018, the  memorandum strongly encourages agencies to implement them  throughout their data centers immediately.
While OMB is primarily responsible for DCOI, its August 2016  memorandum designated the General Services Administration’s (GSA)  Office of Government-wide Policy as a managing partner of the federal  government data center line of business and data center shared  services. More specifically, OMB’s memorandum states that this office is  responsible for, among other things, providing guidance on technology  advancements, innovation, cybersecurity, and best practices to data  center providers and consumers of data center services. Further, the  memorandum states that the office is responsible for assisting with  creating and maintaining an inventory of acquisition tools and products  related to data center optimization, including procurement vehicles for the  acquisition of automated monitoring tools.
section_title: GAO Has Reported and Testified on Issues Related to Consolidating and Optimizing Data Centers
paragraphs: From July 2011 through May 2017, we issued a number of reports and  testified on agency efforts to consolidate and optimize federal data  centers and achieve cost savings. For example, in September 2014, we  reported that, while agencies had made progress on their consolidation  efforts, the total number of data centers reported by agencies had  continued to grow since 2011 as a result of OMB’s expanded definition  and improved inventory reporting. More specifically, we determined that  agencies had collectively reported 9,658 data centers in their  inventories—an increase of about 6,500 compared to OMB’s previous  estimate from December 2011. We noted that agencies had plans to  close about 3,700 data centers by September 2015.
We also reported that 19 of the 24 FDCCI agencies had collectively  reported achieving an estimated $1.1 billion in cost savings for fiscal  years 2011 through 2013, and that, by 2017, that figure was estimated to  rise to about $5.3 billion. However, we pointed out that planned savings  may be higher because 6 agencies—the Departments of Health and  Human Services (HHS), Interior (Interior), Justice (Justice), and Labor  (Labor), GSA, and the National Aeronautics and Space Administration  (NASA)—that reported closing as many as 67 data centers had also  reported limited or no savings.
In addition, our 2014 report noted that 11 of the 21 agencies with planned  cost savings had underreported their fiscal years 2012 through 2015  figures to OMB by approximately $2.2 billion. While several agencies  noted communication issues as the reason for underreporting, others did  not provide a reason. We concluded that, until agencies fully report their  savings, the $5.3 billion in total savings would be understated. Further,  we reported that OMB’s May 2014 data center optimization metrics did  not address server utilization, even though OMB reported this to be as  low as 5 percent across the federal government in 2009. We noted that,  without this metric, OMB may lack important information on agencies’  progress. As a result, we recommended that it implement a metric for  server utilization and assist six agencies in reporting their consolidation  cost savings; we also recommended that agencies fully report their  consolidation cost savings. OMB and the agencies to which we made  recommendations generally agreed with them. OMB subsequently  established a metric to measure agencies’ server utilization progress in its  August 2016 memorandum.
In March 2016, we reported that agencies had continued to make  progress in their data center consolidation efforts. Specifically, we noted  that agencies had reported closing 3,125 of the 10,584 total data centers  as of November 2015. We further noted that 19 of the 24 agencies had  reported achieving an estimated $2.8 billion in cost savings and  avoidances from their data center consolidation and optimization effort for  fiscal years 2011 through 2015. Agencies were also planning an  additional $5.4 billion in cost savings and avoidances, for a total of  approximately $8.2 billion, through fiscal year 2019. However, we noted  that planned savings may be higher because 10 agencies that reported  planned closures from fiscal years 2016 through 2018 had not fully  developed their cost savings goals for these fiscal years.
In addition, we reported that 22 agencies had made limited progress  against OMB’s fiscal year 2015 data center optimization performance  metrics, such as the utilization of data center facilities. Accordingly, we  recommended that the agencies take actions to complete their cost  savings targets and improve optimization progress. Most agencies agreed  with the recommendations or had no comments.
Finally, in May 2017, we reported that agencies continued to  consolidate their data centers, including closing 4,388 of the 9,995 total  data centers as of August 2016. Figure 2 provides a summary of the total  number of data centers and closures reported from 1998 through August  2016.
However, we pointed out that agency progress in achieving savings had  slowed and planned goals had been reduced. Specifically, 18 of the 24  agencies had reported achieving an estimated $2.3 billion in cost savings  and avoidances from their data center consolidation and optimization  efforts from the start of fiscal year 2012 to August 2016, which was about  $451 million less than the total amount of achieved cost savings and  avoidances that agencies reported to us in November 2015. In addition,  agencies’ total planned cost savings of about $633 million were more  than $3.4 billion less compared to the amounts that agencies reported to  us in November 2015, and more than $2.1 billion less than OMB’s fiscal  year 2018 cost savings goal of $2.7 billion.
Our May 2017 report also identified weaknesses in agencies’ DCOI  strategic plans. Of the 23 agencies that submitted their strategic plans at  the time of our review, 7—the Departments of Agriculture (Agriculture),  Education (Education), Homeland Security (DHS), and Housing and  Urban Development (HUD); GSA; the National Science Foundation  (NSF); and the Office of Personnel Management (OPM)—had addressed  all five required elements of a strategic plan, as identified by OMB (such  as providing information related to data center closures and cost savings  metrics). The remaining 16 agencies either partially met or did not meet  the requirements. We also pointed out that there were inconsistencies in  the reporting of cost savings in the strategic plans of 11 agencies.
We concluded that, until agencies address the weaknesses in their DCOI  strategic plans, they may be challenged in implementing the data center  consolidation and optimization provisions of FITARA. Accordingly, we  recommended that OMB improve its oversight of agencies’ DCOI  strategic plans and their reporting of cost savings and avoidances. We  also recommended that 17 agencies complete the missing elements in  their strategic plans and that 11 agencies ensure the reporting of  consistent cost savings and avoidance information to OMB. Twelve  agencies agreed with our recommendations, 2 disagreed, and 11 did not  state whether they agreed or disagreed. The 2 agencies that disagreed— HUD and the Nuclear Regulatory Commission (NRC)—asserted that they  had submitted complete strategic plans. After further review, we agreed  that HUD had provided a complete plan and removed our  recommendation. However, we determined that NRC’s plan was still  incomplete and maintained that our recommendation was appropriate.
section_title: Agencies Are Reporting Limited Progress against OMB’s Data Center Optimization Targets
paragraphs: As mentioned earlier, FITARA required OMB to establish metrics to  measure the optimization of data centers, including server efficiency, and  ensure that agencies’ progress toward meeting the metrics is made  publicly available. Pursuant to FITARA, OMB’s August 2016  memorandum established a set of five data center optimization metrics  intended to measure agency’s progress in the areas of server utilization  and automated monitoring, energy metering, power usage effectiveness,  facility utilization, and virtualization. According to OMB, the server  utilization and automated monitoring metric applies to agency-owned  tiered and non-tiered data centers, while the four remaining metrics apply  to agency-owned tiered centers only.
OMB’s memorandum also established a target value for each of the five  metrics, which agencies are expected to achieve by the end of fiscal year  2018. OMB measures agencies’ progress against the optimization targets  using the agencies’ quarterly data center inventory submission and  publicly reports this progress information on its Dashboard. Table 1  provides a description of the data center optimization metrics and target  values that agencies are expected to achieve by the end of fiscal year  2018.
As of February 2017, 22 of the 24 DCOI agencies reported limited  progress against OMB’s fiscal year 2018 data center optimization targets  on the Dashboard. The remaining 2 agencies—Education and HUD— reported that they did not have any agency-owned data centers in their  inventory and, therefore, did not have a basis to measure and report  optimization progress.
With regard to the data center optimization targets, the most progress  was reported for the power usage effectiveness and virtualization metrics,  with 5 agencies reporting that they had met OMB’s targets. However, 2  agencies or less reported meeting the target for energy metering, facility  utilization, and server utilization and automated monitoring. Figure 3  summarizes the 24 agencies’ progress in meeting each optimization  target, as of February 2017. Following the figure is a more detailed  discussion of the progress of each of the 24 agencies.
Among the 24 agencies, SSA and EPA reported the most progress by  meeting three targets, 20 reported meeting one or none of the targets,  and the remaining 2 agencies did not have a basis to report on progress  because they did not have any agency-owned data centers. Of the 22  agencies reporting progress information, 9 were not able to report  progress against either the server utilization metric or power usage  effectiveness metric, or both, because they lacked the required  monitoring tools to measure progress in these areas. OMB began  requiring the implementation of these monitoring tools in August 2016;  however, as of February 2017, these 9 agencies were not yet reporting  implementation of the tools at any of their data centers. This issue is  discussed in greater detailed later in this report. Table 2 lists the agencies  that met or did not meet each OMB target.
Agencies’ limited progress against OMB’s optimization targets is due, in  part, to them not fully addressing our prior recommendations in this area.  As noted earlier, in March 2016, we reported on weaknesses in  agencies’ data center optimization efforts, including that 22 agencies did  not meet OMB’s fiscal year 2015 optimization targets. We noted that  this was partially due to the agencies facing challenges in optimizing their  data centers, including their decentralized organizational structures that  made consolidation and optimization difficult and competing priorities for  resources. In addition, consolidating certain data centers was problematic  because the volume or type of information involved required the data  center to be close in proximity to the users. Accordingly, we  recommended that the agencies take action to improve optimization  progress, to include addressing any identified challenges. Most agencies  agreed with our recommendations or had no comments.
In response to our recommendation, 19 of the 22 agencies submitted  corrective action plans to us that described steps they intended to take to  improve their data center optimization efforts. Among these steps were  developing internal scorecards to track and report on optimization  progress, including progress at their component agencies, and launching  more aggressive efforts to optimize data centers using virtualization and  cloud computing solutions. While 2 of the 22 agencies—Education and  HUD—are no longer subject to OMB’s optimization metrics based on  OMB’s August 2016 memorandum and their current data center  inventory, none of the remaining 20 agencies had fully addressed our  recommendation as of May 2017.
The importance of overcoming optimization challenges and addressing  our prior recommendations is critical to the ability of agencies to  implement the data center optimization provisions of FITARA and achieve  OMB’s fiscal year 2018 optimization targets. Going forward, it will be  important for the 19 agencies that have established corrective action  plans to continue to execute them and monitor the impact of actions  completed on their optimization progress. Until agencies fully implement  our prior recommendations to address their challenges and improve  optimization progress, they may be hindered in implementing the data  optimization provisions of FITARA and OMB guidance intended to  increase operational efficiency and achieve cost savings. Further, OMB  may be challenged in demonstrating that DCOI is meeting its established  objectives.
subsections: section_title: Most Agencies Are Not Planning to Meet OMB’s Fiscal Year 2018 Optimization Targets
paragraphs: In addition to reporting current optimization progress on the Dashboard,  OMB requires agencies’ DCOI strategic plans to include, among other  things, planned performance levels for fiscal years 2017 and 2018 for  each optimization metric.
However, according to the 24 agencies’ DCOI strategic plan information  as of April 2017, most are not planning to meet OMB’s optimization  targets by the end of fiscal year 2018. More specifically, of the 24  agencies, 5—the Department of Commerce (Commerce), EPA, NSF, the  Small Business Administration (SBA), and the U.S. Agency for  International Development (USAID)—reported plans to fully meet their  applicable targets by the end of fiscal year 2018; 13 reported plans to  meet some, but not all, of the targets; 4 reported that they do not plan to  meet any targets; and 2 do not have a basis to report planned  optimization milestones because they do not report having any agency- owned data centers. Figure 4 summarizes agencies’ progress in meeting  OMB’s optimization targets as of February 2017, and planned progress to  be achieved by September 2017 and September 2018, as of April 2017.
Agencies’ reported plans to meet the optimization targets also vary by  metric. Specifically, about half of the 22 agencies reported plans to meet  the facility utilization and virtualization metrics by the end of fiscal year  2018, while less than half are planning to meet the server utilization and  automated monitoring, energy metering, and power usage effectiveness  metrics. Further, agencies reported that they plan to make the least  amount of progress in meeting the target for power usage effectiveness.  Figure 5 provides a summary, by optimization metric, of agencies’ current  progress in meeting the targets as of February 2017, and planned  progress to be achieved by September 2017 and September 2018, as of  April 2017.
The limited progress made by agencies in optimizing their data centers,  combined with the lack of established plans to improve progress, makes it  unclear whether agencies will be able to achieve OMB’s optimization  targets by the end of fiscal year 2018. Considering that OMB is expecting  at least $2.7 billion in cost savings from agencies’ optimization efforts, the  ability of agencies to meet the optimization targets will be critical to  meeting this savings goal. However, with less than 2 years remaining until  OMB’s fiscal year 2018 DCOI optimization target deadline and the  expiration of the data center consolidation and optimization provisions of  FITARA in October 2018, only five agencies are planning to meet all of  their applicable targets. With the majority of agencies not planning to  meet the optimization targets, there is an increased likelihood that  agencies will need more time beyond 2018 to continue to implement their  optimization efforts. Extending the data center consolidation and  optimization provision of FITARA beyond the current October 2018  horizon could provide agencies with additional time to realize the benefits  of optimization, including cost savings.
section_title: Agencies Reported Successes and Challenges in Optimizing Data Centers
paragraphs: The 24 DCOI agencies reported successes in optimizing their data  centers—notably, the benefits of key technologies, such as virtualizing  systems to improve performance, and increased energy efficiency.  However, agencies also reported operational, technical, and financial  challenges related to, for example, improving the utilization of their data  center facilities, measuring server utilization, and obtaining funding within  their agency for optimization efforts. It will be important for agencies to  take action to address their identified challenges—as we previously  recommended—in order to improve data center optimization progress.
subsections: section_title: Agencies Identified the Use of Optimization Technologies and Improved Energy Efficiency as the Most Reported Successes
paragraphs: Agencies reported a variety of successes in optimizing data centers.  Specifically, the 24 agencies reported a total of 23 areas of success.  Eight areas of successes were identified by three or more agencies, with  the most reported successes for an area being identified by 17 agencies.  The two most reported areas of success—implementing virtualization  technologies and migrating IT applications and services to cloud  computing solutions—were similar to the top reported success in  achieving consolidation cost savings that we identified in 2014 (i.e.,  focusing on virtualization and cloud services as consolidation solutions).  Agencies are also continuing to report successes in other areas that we  highlighted in 2014, including improved energy efficiency, standardized  technology, and improved data center inventory reporting. Table 3 details  the reported areas of success, as well as the number of related agencies.  The most common areas of success are further discussed after the table.
Seventeen agencies reported that implementing virtualization  technologies (i.e., running multiple, software-based machines with  different operating systems on the same physical machine) has proven  successful in optimizing their data centers. For example, officials from  Commerce’s Office of the CIO stated that the department had made the  most notable optimization progress in virtualizing all non-high  performance computing servers, including approximately 11,700  operating systems (as of October 2016). Additionally, officials from  Labor’s Office of the CIO noted that virtualization had helped the  department create a highly efficient, lower cost, common operating  environment suitable for hosting mission-critical applications and  services. The officials added that the department expects to significantly  increase its migration activity and closures in fiscal years 2017 and 2018  by leveraging the portability of this highly virtualized environment.
As another example, officials from GSA’s IT office stated that the agency  has achieved success in retiring older physical systems and shifting to  newer, virtualized technologies. The officials stated that these actions  have contributed to greater flexibility, stability, and redundancy in the  agency’s IT capabilities. Further, officials from NRC’s Office of the CIO  stated that their agency had virtualized 72 percent of its servers, which  allowed the agency to significantly reduce the amount of old, outdated,  and energy-inefficient equipment. Thirteen other agencies also stated that  implementing virtualization technologies had led to successes in  optimizing their data centers.
subsections: section_title: About Half of the Agencies Reported That Migrating IT Applications and Services to Cloud Computing Solutions Increased Optimization Progress
paragraphs: Thirteen agencies reported that migrating IT applications and services to  cloud computing solutions had led to successes in optimizing their data  centers. For example, officials from HHS’s Office of the CIO stated that  one of the department’s offices had realized substantial value with the  use of cloud-provided solutions, including reducing the cost of data center  services by approximately 15 percent compared to government and on- premises data centers.
Additionally, officials from NSF’s Office of Information and Resource  Management stated that the agency successfully reduced and  streamlined its IT footprint through a number of different efforts, such as  migration of applications, e-mail, and instant messaging to cloud  providers; networking technology standardization; and server and storage  consolidation.
As another example, officials from USAID’s Office of the CIO stated that  in 2011 the agency transformed and migrated its primary data center to a  private infrastructure cloud provider, thereby eliminating physical  infrastructure issues (e.g., power, heating, ventilation, air conditioning,  and physical security issues). The officials added that the cloud solution  provided the data center infrastructure, network access, connectivity, and  other services needed to ensure the delivery of critical business services.
Further, officials from Interior’s Office of the CIO stated that the  department had migrated 70,000 users off of 14 legacy e-mail systems to  a single department-wide cloud-based e-mail communications and  collaboration system. Nine other agencies also stated that migrating to  cloud computing solutions led to successes in optimizing their data  centers. Their reported successes ranged from migrating e-mail  applications to the cloud solutions to responding more timely to shifts in  user demand.
section_title: Five Agencies Reported Success in Increasing Energy Efficiency to Optimize Their Data Centers
paragraphs: Five agencies reported that increasing their energy efficiency had led to  success in optimizing their data centers. For example, officials from the  Department of State’s (State) Bureau of Information Resource  Management noted that the department has had success in deploying  modular data centers that utilize energy-efficient power systems and  other optimized operating features that help to reduce the department’s  carbon footprint. Further, a program manager from the SSA’s Office of  Hardware Engineering stated that the agency had improved its energy  efficiency and reduced its carbon footprint through various initiatives  including, among other things, rainwater reclamation, improved  monitoring of IT equipment power usage, energy-efficient lighting, and the  use of solar panels. Figure 6 shows the use of solar panels at the SSA’s  National Support Center.
Additionally, officials from EPA’s Office of Environmental Information  stated that the agency had success in improving energy efficiency  through the purchase of energy efficient IT equipment and by including  energy metering in data center facilities planning and buildout to assist  with validating energy optimization metrics. The officials also noted that  the agency had increased the operating temperature in some data  centers as well as used alternate methods of cooling (e.g., outdoor air to  cool its data centers), which helped the agency improve its energy  efficiency. Officials from Commerce and HHS further stated that  increasing their energy efficiency by, for example, purchasing energy- efficient equipment and deploying power monitoring equipment, had led  to successes in optimizing their data centers.
section_title: Agencies Reported Operational, Technical, and Financial Challenges in Optimizing Their Data Centers
paragraphs: Agencies also reported facing a variety of challenges in optimizing their  data centers. Specifically, the 24 DCOI agencies identified a total of 27  types of challenges across three areas: operational, technical, and  financial. The highest number of challenges were reported in the  operational and technical areas, which included improving data center  facility utilization and measuring and reporting on server utilization.
Certain challenges reported were similar to those described to us by  agencies in 2016, including those related to competing priorities for  labor resources and closing data centers that provide mission critical  applications that require proximity to users. Agencies also continued to  report operational, technical, and financial challenges that were similar to  those described to us in 2014, including gathering data from component  agencies, determining power usage information, and obtaining funding  from within their agency. For example, in 2014, six agencies noted that  gathering data from component agencies was an operational challenge to  achieving consolidation cost savings; however, only two agencies are  now reporting that as a challenge in optimizing their data centers.
Agencies also cited many new challenges that are specific to optimizing  their data centers, such as incorporating enterprise-wide efficiencies  when data centers are owned and managed by multiple organizations  and the significant upfront costs required to purchase data center  monitoring tools. Table 4 details the reported challenges in optimizing  data centers, as well as the number of related agencies. The most  common challenges are further discussed after the table.
Agencies reported the most operational challenges in the following areas:  improving data center facility utilization; competing priorities for labor  resources with other agency IT efforts; shifting definitions of a data center  and changes to data center optimization requirements; and incorporating  enterprise-wide efficiencies when data centers are owned and managed  by multiple organizations.
Improving data center facility utilization: Nine agencies cited this  challenge. For example, officials from the Department of Veterans  Affairs’ (VA) Infrastructure Operations stated that increasing  virtualization generally reduces the number of active server racks in  the space and, therefore, decreases facility utilization. The officials  added that, for smaller rooms that are part of a larger, agency-owned,  multi-functional facility, reducing the size of the room is most often not  an economical decision, as it does not lead to energy savings or  reduced facility costs but, instead, moves the recurring cost of the  space from IT to other functions. Officials from DHS, Interior, Labor,  and NRC also reported that their increased use of virtualization has  negatively impacted their ability to increase facility utilization.
As another example of a challenge in improving facility utilization,  officials from Commerce’s Office of the CIO stated that the National  Oceanic and Atmospheric Administration’s weather field office data  centers contain systems that are proprietary and connect to local  weather sensing instruments or satellite communication equipment.  The officials said that most of these data centers are averaging only  50 percent facility utilization and have no plans to increase, but are  difficult to close because they contain systems designed specifically  for the agency’s mission. Officials from Agriculture, GSA, NASA, and  the Department of the Treasury (Treasury) also cited challenges in  improving facility utilization.
Competing priorities for labor resources with other agency IT efforts:  Six agencies cited this challenge. For example, officials from  Commerce’s Office of the CIO stated the Census Bureau is ramping  up for a very large program—the 2020 Decennial Census—while also  working to optimize its data centers. This has led to challenges in  implementing data center infrastructure management tools and  replacing old power distribution units with new ones. As another  example, officials from EPA’s Office of Environmental Information  noted that IT personnel are primarily focused on day-to-day  operations and maintenance activities and, therefore, resources  normally used to support data center activities are periodically pulled  away to address more immediate operational activities (such as  cybersecurity initiatives).
In addition, SBA’s CIO stated that the biggest challenge faced by the  agency is a lack of labor resources, which has historically been due to  a focus on mission priorities instead of data center improvements.  Officials from GSA, Labor, and Transportation also stated that  completing priorities for labor resources has been a challenge to  optimizing their data centers.
Shifting definition of a data center and changes to data center  optimization requirements: Five agencies cited this challenge. For  example, officials from Interior’s Office of the CIO stated that  significant changes outlined in OMB’s August 2016 memorandum  and previously issued guidance related to the definitions of a data  center and optimization metrics presented challenges in maintaining  inventories, measuring progress, and assessing cost savings and  avoidances. As another example, officials from Energy’s Office of the  CIO stated that Energy’s unique computing environments, which  support scientific research, facility and plant operations, power  management, and mission-specific computing, makes aligning with  OMB’s data center definition difficult.
In addition, officials from DHS’s Office of the CIO stated that OMB’s  recent changes to the data center optimization metrics, including the  focus on agency-owned data centers, greatly impacted the  department’s ability to report on optimization progress. More  specifically, the officials stated that OMB’s prior optimization metrics  focused on the department’s three core data centers (i.e., primary  consolidation points); however, under OMB’s new metrics, the  department’s core data centers are no longer applicable to the metrics  because they are not agency-owned. The officials added that this  negatively impacted the department’s ability to report optimization  progress related to power usage effectiveness. Officials from GSA  and NRC also cited challenges related to the change in the definition  of a data center and data center optimization requirements.
Incorporating enterprise-wide efficiencies for data centers owned and  managed by multiple organizations: Five agencies identified this  challenge. For example, officials from NASA’s Office of the CIO stated  that, historically, the agency’s data centers have been owned and  managed by multiple organizations, including contractors, which has  made it challenging to incorporate enterprise-enabled efficiencies (i.e.,  common procurements, implementation of standard hardware,  software, and management tools). The officials also mentioned that  the extensive use of data centers collocated within multi-use  buildings, with shared electrical and mechanical infrastructure, has  resulted in the agency not realizing the magnitude of savings that  would be attributed to the closure of stand-alone data center facilities.
As another example, officials in Energy’s Office of the CIO stated that  the implementation of optimization solutions in data centers that are  mission and research specific, or have unique operational and  environmental requirements, has presented operational challenges. In  addition, officials from Justice’s Office of the CIO stated that  implementing enterprise solutions across a large and traditionally  federated organization has been challenging. Officials from DHS and  Labor also cited challenges with improving optimization at data  centers that are owned and managed by multiple organizations.
subsections: section_title: Technical Challenges
paragraphs: Agencies reported the most technical challenges in the following areas:  measuring and reporting on server utilization progress, a lack of electricity  metering to determine power usage information, and poor network  connectivity and low bandwidth at field locations constraining  consolidation and optimization efforts.
Measuring and reporting on server utilization progress: Nine agencies  cited this challenge. For example, officials from VA’s Infrastructure  Operations cited challenges with the complexity of programming the  tools needed to collect the data to measure server utilization. In  particular, the officials noted issues in delineating what data should be  collected to determine server “busy” and “idle” times (e.g., computer  processing unit usage, power consumption, or other data) and what  unit of time to associate with the data collection (i.e., seconds,  minutes, hours, etc.) in order to be able to report on the server  utilization metric. As another example, officials from Justice’s Office of  the CIO stated that optimizing the server utilization of department data  centers that are consolidation points will be extremely difficult  because the environments are going through significant changes as  they receive servers from other locations.
In addition, officials from Treasury’s Office of the CIO stated that,  while the department’s servers have the ability to measure and  monitor processing usage, most data centers do not have the ability to  centrally aggregate and report on that data. Officials from Agriculture,  the Department of Defense (Defense), Labor, NASA, OPM, and  Treasury also cited challenges in measuring and reporting on server  utilization progress.
Lack of electricity metering to determine power usage information:  Seven agencies identified this challenge. For example, officials from  Commerce’s Office of the CIO stated that many of the department’s  data centers are small and lack separate power metering. The officials  added that, rather than adding power monitoring to each small data  center, the department needs to conduct further research to evaluate  whether consolidation of these unmetered data centers into a few  larger well-maintained data centers is more cost effective. As another  example, officials from Labor’s Office of the CIO stated that a vast  majority of the department’s data centers are in modified office spaces  that also serve other purposes, such as accommodating the storage  of legacy IT assets and providing a workspace for IT support  personnel, which has made the installation of power metering  challenging.
Further, officials from VA’s Infrastructure Operations stated that the  department’s individual data centers are largely unique, thus requiring  detailed engineering to determine how to retrofit energy metering  solutions to provide the data necessary for energy usage optimization,  particularly without incurring critical IT system downtime. The officials  added that the majority of the department’s data centers are not  stand-alone data centers, but rather, are rooms within a medical  center facility or other multi-purpose facility that were not constructed  to facilitate power metering. VA officials stated that these challenges  made measuring power usage effectiveness extremely complicated,  time-consuming, and costly. Agriculture, Labor, OPM, and SBA also  mentioned challenges related to the lack of electricity metering to  determine power usage information.
Poor network connectivity and low bandwidth at field locations  constrains consolidation and optimization efforts: Five agencies cited  this challenge. For example, officials from Interior’s Office of the CIO  stated that numerous remote field offices within the department  experience poor network connectivity and low bandwidth to support  running remotely-hosted applications. The officials added that the risk  of reduced service levels at these remote locations is frequently cited  as a constraint on consolidation and a challenge to improving  optimization progress. As another example, Transportation’s Office of  the CIO noted challenges with consolidating field site servers because  the telecommunication bandwidth to the field sites is lacking. Officials  from HHS, Labor, and SBA also cited concerns about connectivity  performance issues as a challenge to consolidation and optimization  of data centers at their field office locations.
section_title: Financial Challenges
paragraphs: Agencies reported three financial challenges in the following areas:  obtaining the funding within their agency for optimization efforts, the  upfront costs required to purchase the monitoring tools needed to  measure optimization progress, and determining the resulting cost  savings and avoidances.
Obtaining the funding within their agency for optimization efforts: Ten  agencies cited this challenge. For example, officials from OPM’s  Office of the CIO stated that while the agency’s base budget includes  ongoing operations and maintenance funding for the agency’s existing  data centers, the availability of financial resources during fiscal years  2017 and 2018 would be one of the most significant challenges to  improving data center optimization performance, and satisfying DCOI  requirements. Further, officials from Justice’s Office of the CIO stated  that financial constraints may limit the funding available for migration  of component infrastructure to cloud computing services or the  department’s core enterprise facilities, which could delay or prevent  optimization.
As another example, officials from Defense’s Office of the CIO stated  that resource constraints to support application and system  rationalization, re-engineering, and migration, forced many component  agencies to focus on physical relocations of systems, which limit data  center optimization opportunities and savings. Officials from  Commerce, DHS, Energy, HHS, Labor, SBA, and Transportation also  cited challenges in obtaining the funding within their agency for  optimization efforts.
Significant upfront costs required to purchase the monitoring tools  needed to measure optimization progress: Eight agencies cited this  challenge. For example, officials from Treasury’s Office of the CIO  stated that the department is currently in the process of evaluating  how to most effectively meet data center power metering  requirements without incurring significant expenditures. The officials  stated that several of their larger data centers are in older, multi-use  buildings and share a cooling infrastructure with the entire building.  The officials added that measuring the energy consumed by the  portions of the building dedicated to hosting IT equipment would  require meters to be installed within just those spaces dedicated to IT,  which is a significant cost that is being evaluated relative to other  mission-oriented investments.
As another example, officials from Interior’s Office of the CIO stated  that the investment for purchasing data center optimization tools  would require a reallocation of funds from the department’s fiscal  years 2017 and 2018 budgets and would have an adverse effect on  meeting other higher priority requirements, such as cybersecurity  requirements. Further, the officials stated that purchasing and  deploying energy metering tools in the department’s smaller data  centers would result in a negative return on investment. In addition,  officials from VA’s Infrastructure Operations stated that the  department’s data centers are largely unique and require detailed  engineering to determine how to retrofit metering solutions to provide  data necessary for energy usage optimization, which has not yet been  funded. Officials from Agriculture, Commerce, Defense, GSA, and  State also cited significant upfront costs of data center monitoring  tools as a challenge.
Determining the resulting cost savings and avoidances from  consolidation and optimization efforts: Five agencies identified this  challenge. For example, officials from NASA’s Office of the CIO stated  that due to their extensive use of data centers collocated within multi- use buildings, with shared electrical and mechanical infrastructure, the  agency has not realized the magnitude of savings that would be  attributed to the closure of stand-alone data center facilities. The  officials added that, in most instances, the closed data center spaces  have been locally repurposed for non-IT use.
As another example, officials from Agriculture’s Office of the CIO  stated that it can be difficult to determine facility costs and the  resulting cost savings and avoidances. The officials noted that data  centers located within government owned or leased buildings usually  do not pay for electricity, heating and air conditioning expenses, or  lease and facility upkeep costs, which can present challenges in  calculating any cost savings and avoidances from optimization.  Officials from GSA, Interior, and Treasury also cited challenges in  determining the resulting cost savings and avoidances from their  consolidation and optimization efforts.
Addressing these optimization challenges and others—as we previously  recommended in 2016—is increasingly important in light of FITARA’s  requirements, which direct agencies to establish a multi-year strategic  plan to improve data center optimization progress. Until agencies address  these challenges, they could be hindered in the implementation of their  data center optimization strategic plans and in making initiative-wide  progress against OMB’s optimization targets.
section_title: Most Agencies Still Cannot Effectively Measure Server Utilization
paragraphs: As noted earlier, FITARA required OMB to establish data center  consolidation and optimization metrics, including a metric specific to  measuring server efficiency; it also required agencies to report on  progress in meeting the metrics. Pursuant to FITARA, OMB’s August  2016 memorandum required agencies to measure and report on server  utilization progress, including the number of agency-owned data centers  fully equipped with automated monitoring tools and their server utilization  percentages.
To effectively measure progress against this metric, OMB’s memorandum  also directed agencies to immediately begin replacing the manual  collection and reporting of systems, software, and hardware inventory  housed within agency-owned data centers with automated monitoring  tools and to complete this effort no later than the end of fiscal year 2018.  Agencies are required to report progress in implementing automated  monitoring tools and server utilization averages at each data center as  part of their quarterly data center inventory reporting to OMB. Finally,  standards for internal control emphasize the need for federal agencies to  establish plans to help ensure goals and objectives can be met, including  compliance with applicable laws and regulations.
As of February 2017, 4 of the 22 agencies reporting agency-owned data  centers in their inventory—NASA, NSF, SSA, and USAID—reported  that they had implemented automated monitoring tools at all of their data  centers. Further, 10 reported that they had implemented automated  monitoring tools at between 1 and 57 percent of their centers, and 8 had  not yet begun to report the implementation of these tools. In total, the 22  agencies reported that automated tools were implemented at 123 (or  about 3 percent) of the 4,528 total agency-owned data centers, while the  remaining 4,405 (or about 97 percent) of these data centers were not  reported as having these tools implemented. Table 5 provides a listing of  the number and related percentage of agency-owned data centers  reported by agencies as having automated monitoring tools implemented.
Of the 123 data centers reported as having automated monitoring tools  implemented, 59 were identified as tiered data centers and 64 as non- tiered data centers. Figure 7 summarizes the number of agency-owned  data centers reported with automated monitoring tools installed, including  the number of tiered and non-tiered centers.
The limited implementation of automated monitoring tools resulted in  incomplete information on server utilization percentages. As noted earlier,  OMB’s IT Dashboard is used to publicly report on agencies’ progress in  measuring server utilization. This progress information is obtained from  agencies’ quarterly data center inventory submissions, which are required  to include detailed data on the server utilization averages of each tiered  and non-tiered data center. Based on agencies’ February 2017 data  center inventory data, 4 of the 22 agencies reported a server utilization  average for all of their monitored tiered and non-tiered data centers, 10  reported server utilization averages at a portion of their centers, and 8 did  not report this information. SSA reported the highest server utilization  average of 100 percent at its one agency-owned tiered data center,  while GSA reported the lowest percentage of 9 percent across its 31  agency-owned tiered and non-tiered centers with automated monitoring  tools installed.
According to our analysis of agencies’ inventory data, the average server  utilization across all 123 data centers with automated monitoring tools  installed was about 28 percent, which is approximately 37 percent below  OMB’s fiscal year 2018 goal of 65 percent or higher. Figure 8 shows the  agency-reported server utilization averages for the 4 agencies that  reported this information at all their data centers and the 10 agencies that  reported this information at a portion of their centers, as well as the  percentage of their agency-owned data centers with automated  monitoring tools installed.
For the 18 agencies that did not report server utilization average  information at all their data centers, none fully documented plans to  implement the automated monitoring tools required to measure this  information at all their agency-owned tiered and non-tiered centers by the  end of fiscal year 2018. More specifically, our analysis of agencies’ DCOI  strategic plans, FITARA implementation milestones, and other  documentation (such as project plans and charters) showed that 6 of the  18 agencies—Agriculture, Energy, EPA, GSA, State, and VA—partially  documented plans because they addressed implementing automated  monitoring tools for only a portion of their data centers. However, these  agencies did not address implementing such tools at all tiered and non- tiered agency-owned data centers, as required by OMB. The remaining  12 agencies did not document plans to implement automated monitoring  tools. Table 5 provides an assessment of agencies’ documented plans to  implement data center automated monitoring tools.
The 18 agencies provided a variety of reasons regarding why they had  not established a plan to implement automated monitoring tools at all  agency-owned data centers. For example, officials at six agencies  (Defense, DHS, EPA, GSA, Labor, and Justice) stated that they were in  the process of establishing a plan to implement automated monitoring  tools, but had not yet completed it. As another example, agency officials  from State’s Bureau of Information Resource Management and NRC’s  Office of the CIO noted that they were still evaluating options for  purchasing and deploying these tools. Further, officials from OPM’s Office  of the CIO and Transportation’s Office of the CIO stated that they were  still determining the extent to which their data centers had automated  monitoring tools installed. Lastly, officials from Commerce’s Office of the  CIO stated the department had no specific plans to invest in automated  monitoring tools.
The lack of detailed plans to implement automated monitoring tools at all  agency-owned data centers is also due, in part, to OMB not having  established a formal requirement to document such plans. Although  OMB’s August 2016 memorandum required agencies to submit a DCOI  strategic plan by September 30, 2016, and to update it by April 14, 2017,  these plans were not required to include detailed information describing  how the agency was planning to meet OMB’s requirement to implement  automated monitoring tools at all agency-owned tiered and non-tiered  centers.
Recognizing this issue, OMB staff from the Office of the Federal CIO  stated that they have been advising agencies to include these more  detailed plans and milestones for implementing data center automated  monitoring tools as part of their publicly available FITARA implementation  milestones. However, OMB has not established a formal requirement in  its data center guidance or FITARA implementation guidance provided to  agencies. As mentioned previously, our analysis of agency’s FITARA  implementation milestones showed that most agencies were not aware of  OMB’s request to include this information.
Until OMB requires agencies to include detailed plans to implement  automated monitoring tools in their FITARA implementation milestones,  agencies may continue to lack a roadmap to meet a key DCOI goal.  Further, until agencies complete their plans, they may be challenged in  implementing the tools needed to effectively measure server utilization—a  data center optimization area highlighted in FITARA, and that we  previously reported as being critical to improving the efficiency,  performance, and environmental footprint of federal data center activities.
section_title: Conclusions
paragraphs: With the August 2016 launch of DCOI, OMB took a considerable step  forward in providing guidance for the implementation of the data center  consolidation and optimization requirements of FITARA and increasing  the oversight of agencies’ efforts to optimize their data centers. OMB’s  fiscal year 2018 optimization targets provide clear and transparent goals  for agencies’ optimization efforts; however, agencies reported limited  progress against those targets. Additionally, although agencies’ DCOI  strategic plans provide a mechanism for agencies to report planned fiscal  years 2017 and 2018 milestones toward achieving OMB’s optimization  targets, most agencies reported that they are not planning to meet OMB’s  targets by the end of fiscal year 2018. Considering that OMB established  a DCOI-wide savings goal of $2.7 billion, the ability of agencies to meet  the optimization targets will be critical to achieving these savings.  Extending the time frame for the agencies to meet the required data  center consolidation and optimization provisions of FITARA beyond  October 2018 could provide agencies with additional time to achieve the  benefits of optimization. In addition, agencies’ implementation of our prior  recommendations to address optimization challenges and improve  progress could help ensure that they are better positioned to meet key  DCOI goals.
As a result of OMB’s increased focus on data center optimization  beginning in 2013 and its more recent efforts to launch DCOI, agencies  have reported noteworthy successes in optimizing their data centers— particularly in leveraging virtualization and cloud computing as a means to  optimize their data centers. These constructive experiences indicate that  DCOI is moving in the right direction. However, as agencies work toward  achieving OMB’s fiscal year 2018 optimization targets, many are reporting  challenges related to improving data center facility utilization, measuring  and reporting on server utilization progress, and obtaining the funding  within their agency for optimization efforts. Such a dynamic environment  reinforces the need for agencies to address their identified challenges— as we previously recommended—in order to improve data center  optimization progress.
OMB’s efforts to establish a metric to measure server utilization as part of  its August 2016 memorandum were consistent with our 2014  recommendation and an important step toward ensuring that agency  computing resources are being used more efficiently. Additionally, OMB’s  requirement that agencies implement automated monitoring tools at their  data centers by the end of fiscal year 2018 will help to ensure that they  have the necessary foundation in place to effectively measure and report  on server utilization progress. However, with agencies collectively  reporting that these tools are only installed at about 3 percent of the total  data centers and with 18 agencies lacking complete plans to implement  these tools at their remaining data centers, significant work remains  toward meeting OMB’s requirement. The lack of a formal OMB  requirement to establish detailed plans in this area and report them to  OMB further increases the likelihood that agencies will continue to lack  them.
In the absence of such a requirement and completed plans, agencies will  be missing an important roadmap for implementing the automated  monitoring tools needed to measure server utilization—an area that both  we and OMB have reported as critical to improving the efficiency,  performance, and environmental footprint of federal data center activities.  Moreover, with automated monitoring tools not required by OMB to be  fully implemented by agencies until the end of fiscal year 2018, extending  the time frame of FITARA’s data center consolidation and optimization  provisions could also better ensure that server utilization is effectively  measured and reported beyond fiscal year 2018, after the necessary  monitoring tools are implemented.
section_title: Matter for Congressional Consideration
paragraphs: As most agencies lack plans to meet OMB’s data center optimization  targets by the end of fiscal year 2018, it is increasingly likely that these  agencies will require additional time to achieve the data center  consolidation and optimization goals required by FITARA and OMB  guidance. In order to provide agencies with additional time to meet OMB’s  data center optimization targets and achieve the related cost savings,  Congress should consider extending the time frame for the data center  consolidation and optimization provisions of FITARA beyond their current  expiration date of October 1, 2018.
section_title: Recommendations for Executive Action
paragraphs: To better ensure that agencies complete important DCOI planning  documentation and that the initiative improves governmental efficiency  and achieves intended cost savings, we are recommending that the  Director of OMB direct the Federal CIO to formally document a  requirement for agencies to include plans, as part of existing OMB  reporting mechanisms, to implement automated monitoring tools at their  agency-owned data centers.
We are also recommending that the Secretaries of Agriculture,  Commerce, Defense, Homeland Security, Energy, HHS, Interior, Labor,  State, Transportation, Treasury, and VA; the Attorney General of the  United States; the Administrators of EPA, GSA, and SBA; the Director of  OPM; and the Chairman of NRC take action to, within existing OMB  reporting mechanisms, complete plans describing how the agency will  achieve OMB’s requirement to implement automated monitoring tools at  all agency-owned data centers by the end of fiscal year 2018.
section_title: Agency Comments and Our Evaluation
paragraphs: We received comments on a draft of this report from OMB and the 24  agencies that we reviewed. Of the 19 agencies to which we made  recommendations, 10 agencies agreed with our recommendations, 3  (Defense, Interior, and OPM) partially agreed, and 6 (including OMB) did  not state whether they agreed or disagreed. In addition, 6 agencies to  which we did not make recommendations stated that they had no  comments. Multiple agencies also provided technical comments, which  we have incorporated as appropriate.
The following discusses the comments from each agency to which we  made a recommendation.
In an e-mail received on July 7, 2017, a staff member from OMB’s  Office of General Counsel stated that the agency had no comments  on the draft report. The staff member did not state whether the agency  agreed or disagreed with our recommendation.
In an e-mail received on June 26, 2017, a senior advisor in the  Department of Agriculture’s Office of the CIO did not state whether the  department agreed or disagreed with our recommendation, but noted  that the department understands that automated monitoring of server  utilization and virtualization is critical to accurate data center  performance and cost savings reporting.
In written comments, Commerce stated that it agreed with our  recommendation and described actions planned to implement it.  Specifically, the department noted that, as part of its effort to  consolidate, define, and establish a plan to deploy an enterprise-wide  automated monitoring tool, it has identified two component agencies  that will offer a data center infrastructure management tool as a  service. The department added that this approach will allow it to  monitor and report cost savings and avoidances more efficiently.  Commerce’s comments are reprinted in appendix II.
In written comments, Defense stated that it partially agreed with our  recommendation. Specifically, the department stated that it  recognizes the value of data center infrastructure management  capabilities in realizing DCOI objectives and will endeavor to  implement the capabilities as quickly as possible. However, the  department noted that it will be unable to complete the implementation  of data center infrastructure management capabilities by the end of  fiscal year 2018, as we recommended. As obstacles to meeting this  deadline, the department cited procurement regulations, resource  challenges, the budget cycle, and remaining work to resolve the  population of installation processing nodes, but did not offer further  details.
Our report specifically recognizes the challenges cited by agencies in  the implementation of automated monitoring tools (i.e., data center  infrastructure management capabilities), and notes the importance of  detailed plans to overcome these challenges. Given the department’s  own acknowledgment of facing implementation obstacles, a plan  describing how it will implement these important monitoring tools  could help overcome the challenges identified. Therefore, we continue  to believe our recommendation is warranted. Defense’s comments are  reprinted in appendix III.
In written comments, Energy stated that the department concurred  with our recommendation and described planned actions to implement  it. Specifically, the department stated that it established plans to  implement automated monitoring tools at its 78 department-owned  tiered data centers and plans to evaluate whether its 68 department- owned non-tiered data centers should be consolidated or closed. For  the non-tiered centers slated to remain open, the department stated  that it expects to complete plans describing how it will automate  server utilization by September 2019. Energy’s comments are  reprinted in appendix IV.
In written comments, HHS stated that the department concurred with  our recommendation and described planned actions to implement it.  Specifically, the department stated that HHS will direct its operating  and staff divisions to acquire and install automated monitoring tools in  all agency-owned data centers by the close of fiscal year 2018. HHS’s  comments are reprinted in appendix V.
In written comments, DHS stated that the department concurred with  our recommendation and described planned actions to implement it.  Specifically, the department stated that it is continually reviewing  optimization alternatives, including evaluating the option to move to a  cloud deployment model over the next few years. The department  further noted that it does not expect to achieve the optimum solution  in agency-owned tiered data centers by the end of fiscal year 2018, as  we recommended, but agreed with our suggestion that the DCOI time  frame be reconsidered. In addition, DHS stated that it expects to have  an optimization plan that includes, among other things, resource  requirements and a schedule to achieve monitoring compliance for  agency-owned tiered data centers by April 2018. DHS’s comments  are reprinted in appendix VI.
In written comments, Interior stated that the department partially  concurred with our recommendation. Specifically, the department  stated that it is committed to completing its plan on schedule, but that  its ability to meet OMB’s requirement to implement automated  monitoring tools at all department-owned data centers by the end of  fiscal year 2018, as we recommended, will depend on many factors  and variables, including the availability of funding and other  resources.
Because of the potential for improved efficiency and cost savings from  data center optimization, as discussed in this report, we believe the  department should devote the necessary resources to ensure that  automated monitoring tools are installed at all department-owned data  centers by the end of fiscal year 2018, as required by OMB.  Therefore, in our view, the recommendation continues to be  warranted. Interior’s comments are reprinted in appendix VII.
In an e-mail received on July 13, 2017, a Justice audit liaison stated  that the department concurred with our recommendation.
In written comments, Labor stated that the department accepted our  recommendation and will incorporate pertinent information in its next  data center consolidation and optimization strategic plan due in April  2018. Labor’s comments are reprinted in appendix VIII.
In written comments, State indicated that the department agreed with  our recommendation and described completed and planned actions to  address it. Specifically, the department stated that it performed an  analysis of tools, including shared services and commercial-off-the- shelf products. The department also stated that it is developing an  acquisition strategy based on its research and is recommending that a  commercially available product would be the best solution to meet  monitoring requirements. Further, the department noted that  additional budgetary resources may be required to support an  enterprise-wide roll-out of automated server monitoring across all  tiered data centers, which may not be available until fiscal year 2019  or later.
As discussed in detail in this report, data center optimization holds the  potential for improved efficiency and cost savings. Consequently, we  encourage the department to devote the necessary resources to  ensure that automated monitoring tools are installed at all department- owned data centers by the end of fiscal year 2018, as required by  OMB. State’s comments are reprinted in appendix IX.
In an e-mail received on July 3, 2017, a deputy director in  Transportation’s Audit Relations and Program Improvement office  stated that the department concurred with our recommendation.
In an e-mail received on July 20, 2017, an audit liaison in Treasury’s  Office of the CIO stated that the department had no comments on the  draft report, and did not state whether the agency agreed or disagreed  with our recommendation.
In written comments, VA stated that it concurred with our  recommendation and noted that it is developing a plan to fully comply  with OMB’s requirement to implement automated monitoring tools at  all agency-owned data centers by the end of fiscal year 2018. The  department added that it expects to complete this plan by November  2017. VA’s comments are reprinted in appendix X.
In written comments, EPA did not state whether the agency agreed or  disagreed with our recommendation, but described planned actions to  implement it. Specifically, the agency detailed plans to address OMB's  requirements, such as leveraging EPA's current investment in a  network monitoring tool and the intent to procure and deploy a data  center infrastructure management tool by the end of fiscal year 2018.  However, EPA also noted that budget cuts may delay the agency's  efforts to fully implement the requirements of DCOI.
As noted earlier, because of the potential efficiency and savings from  data center optimization, we believe EPA should devote the  necessary resources to ensure that automated monitoring tools are  installed at all department-owned data centers by the end of fiscal  year 2018, as required by OMB. EPA's written comments are  reprinted in appendix XI.
In written comments, GSA stated that it agreed with our  recommendation and that it plans to install automated monitoring tools  by the end of fiscal year 2018. GSA’s comments are reprinted in  appendix XII.
In written comments, NRC stated that it was in general agreement  with our findings. The agency did not state whether it agreed or  disagreed with our recommendation, but described actions planned to  address it. Specifically, the agency stated that it plans to install  automated monitoring tools in all of its tiered data centers. The  agency added that it is planning to close its non-tiered data centers.  NRC’s comments are reprinted in appendix XIII.
In written comments, OPM stated that the agency partially concurred  with our recommendation. Specifically, the agency stated that it plans  to consolidate its remaining data centers into two main locations by  the end of fiscal year 2018. OPM further stated that this consolidation  will obviate the need to implement automated monitoring tools at the  data centers that are closing. Finally, the agency noted that it is  implementing automated monitoring tools at the designated core data  centers.
We encourage OPM’s efforts to continue to consolidate its data  centers. However, as mentioned in its comments, OPM’s automated  monitoring tools have not yet been installed at the agency’s core data  centers. Completing a plan describing how the agency will meet  OMB’s requirement to implement automated monitoring tools at these  centers, as we recommended, could better ensure that this important  effort is completed. Therefore, we believe our recommendation is still  warranted. OPM’s comments are reprinted in appendix XIV.
In an e-mail received on July 13, 2017, a program manager in SBA’s  Office of Congressional and Legislative Affairs stated that the agency  had no comments on the draft report, and did not state whether the  agency agreed or disagreed with our recommendation.
In addition to the aforementioned comments, six agencies to which we did  not make recommendations provided the following responses:  In an e-mail received on June 23, 2017, a policy analyst in  Education’s Office of the Secretary/Executive Secretariat stated that  the department had no comments on the draft report.
In written comments, HUD stated that the department had no  comments on the draft report. HUD’s comments are reprinted in  appendix XV.
In an e-mail received on July 14, 2017, a NASA audit liaison stated  that the agency had no comments on the draft report.
In an e-mail received on July 17, 2017, a NSF audit liaison stated that  the agency had no comments on the draft report.
In written comments, SSA stated that the agency had no comments  on the draft report. SSA’s comments are reprinted in appendix XVI.
In an e-mail received on July 12, 2017, an audit liaison in USAID’s  Bureau for Management stated that the agency had no comments on  the draft report.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 22 days from the  report date. At that time, we will send copies to interested congressional  committees, the Director of OMB, Secretaries and agency heads of the  departments and agencies addressed in this report, and other interested  parties. In addition, the report will be available at no charge on GAO’s  website at http://www.gao.gov.
If you or your staffs have any questions on the matters discussed in this  report, please contact me at (202) 512-9286 or pownerd@gao.gov.  Contact points for our Offices of Congressional Relations and Public  Affairs may be found on the last page of this report. GAO staff who made  major contributions to this report are listed in appendix XVII.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to (1) assess agencies’ progress against the Office of  Management and Budget’s (OMB) data center optimization targets, (2)  identify agencies’ notable optimization successes and challenges, and (3)  evaluate the extent to which agencies are able to effectively measure  server utilization.
To assess agencies’ progress against OMB’s data center optimization  targets, we analyzed the February 2017 data center optimization progress  information of the 24 department and agencies (agencies) that  participate in OMB’s Data Center Optimization Initiative (DCOI). This  progress information was obtained from the Information Technology (IT)  Dashboard—an OMB public website that provides information on federal  agencies’ major IT investments. We then compared the agencies’  optimization progress information against OMB’s fiscal year 2018  optimization targets, as documented in its August 2016 memorandum.  Although OMB’s memorandum establishes a single optimization target  value for the server utilization and automated monitoring metric, the  Dashboard displays agencies’ progress for tiered and non-tiered data  centers separately. To report consistently with OMB’s implementation  memorandum, we combined the progress information for tiered and non- tiered data centers into a single assessment in this report.
We also reviewed the 24 agencies’ DCOI strategic plans, as of April  2017, to obtain information regarding their fiscal years 2017 and 2018  plans to meet or not meet OMB’s optimization targets. This  documentation included agencies’ strategic plan information publicly posted on agency-owned digital strategy websites, and additional agency- provided documentation of their data center consolidation and  optimization strategic plans.
To assess the reliability of agencies’ optimization progress information on  OMB’s IT Dashboard, we reviewed the information for errors or missing  data, such as progress information that was not available for certain  metrics. We also compared agencies’ optimization progress information  across multiple reporting quarters to identify any inconsistencies in  agencies progress. We discussed with OMB staff any discrepancies or  potential errors identified to determine the causes or request additional  information. In addition, we interviewed OMB officials to obtain additional  information regarding the steps taken to ensure the reliability of and  validate the optimization data on the Dashboard. We determined that the  data were sufficiently reliable to report on agencies’ optimization  progress.
To assess the reliability of the DCOI strategic plans, we reviewed  agencies’ documentation to identify any missing data or errors. We also  compared the planned data center optimization milestones in agencies’  documentation against current optimization progress information obtained  from the Dashboard. In addition, we reviewed agency chief information  officer statements attesting to the completeness of their DCOI strategic  plan information. Moreover, we obtained written responses from agency  officials regarding the steps taken to ensure the accuracy and reliability of  their strategic plan. We discussed with agency officials any discrepancies  or potential errors identified during our reviews of their strategic plan to  determine the causes or request additional information. As a result of  these efforts, we determined that the agencies’ strategic plan information  was sufficiently reliable for reporting on plans to meet or not meet OMB’s  fiscal year 2018 optimization targets.
To address the second objective, we reviewed the 24 agencies DCOI  strategic plans to identify successes and challenges encountered by  agencies in optimizing their data centers. We also interviewed cognizant  officials at the 24 agencies in order to gather additional information about  their data center optimization successes and challenges. We then  categorized the agency-reported successes and challenges to determine  the ones encountered most often.
To evaluate the extent to which selected agencies are able to effectively  measure server utilization, we analyzed the 24 agencies’ February 2017  data center inventory information. We reviewed the inventory information  to determine the extent to which the agencies reported the  implementation of automated monitoring tools at their data centers to  measure server utilization, as well as the reported server utilization  percentages at those centers. To determine whether agencies had  established detailed plans to meet OMB’s M-16-19 requirement to  implement automated monitoring tools at all agency-owned data centers  by the end of fiscal year 2018, we reviewed agencies DCOI strategic  plans, publicly available milestone information for implementing the  December 2014 IT acquisition reform law, and other planning  documentation provided by agencies (such as project charters and  project plans). We reviewed this documentation to determine the extent to  which agencies documented plans to implement automated monitoring  tools at all their agency-owned data centers by the end of fiscal year  2018, as required by OMB.
To assess the reliability of the agencies’ data center inventories, we  checked for missing data and other errors, such as anomalous server  utilization percentage information. We also compared agencies’ reported  use of automated monitoring tools at their data centers across multiple  reporting quarters to identify any inconsistencies in agencies’ progress.  We discussed with agency officials any discrepancies or potential errors  identified to determine the causes or request additional information.  Further, we obtained written responses from agency officials regarding  actions taken to ensure the reliability of their inventory data. We  determined that the agencies’ data were sufficiently reliable to report on  agencies’ progress in implementing automated monitoring tools to  measure server utilization.
We conducted this performance audit from July 2016 to August 2017 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Comments from the Department of Commerce
section_title: Appendix III: Comments from the Department of Defense
section_title: Appendix IV: Comments from the Department of Energy
section_title: Appendix V: Comments from the Department of Health and Human Services
section_title: Appendix VI: Comments from the Department of Homeland Security
section_title: Appendix VII: Comments from the Department of the Interior
section_title: Appendix VIII: Comments from the Department of Labor
section_title: Appendix IX: Comments from the Department of State
section_title: Appendix X: Comments from the Department of Veterans Affairs
section_title: Appendix XI: Comments from the Environmental Protection Agency
section_title: Appendix XII: Comments from the General Services Administration
section_title: Appendix XIII: Comments from the Nuclear Regulatory Commission
section_title: Appendix XIV: Comments from the Office of Personnel Management
section_title: Appendix XV: Comments from the Department of Housing and Urban Development
section_title: Appendix XVI: Comments from the Social Security Administration
section_title: Appendix XVII: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, individuals making contributions  to this report included Dave Hinchman (Assistant Director), Jon Ticehurst  (Assistant Director), Chris Businsky, Rebecca Eyler, Linda  Kochersberger, and Jonathan Wall.