Data Center Consolidation: Agencies Need to Complete Inventories and Plans to Achieve Expected Savings
Published: 2011-07-19T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Over time, the federal government's demand for information technology has led to a dramatic rise in the number of federal data centers and an increase in operational costs. Recognizing this increase, the Office of Management and Budget (OMB) has launched a governmentwide initiative to consolidate data centers. GAO was asked to (1) assess whether agency consolidation documents include adequate detail for agencies to consolidate their centers, (2) identify the key consolidation challenges reported by agencies, and (3) evaluate whether lessons learned during state government consolidation efforts could be leveraged at the federal level. To address these objectives, GAO assessed the completeness of agency inventories and plans, interviewed agencies about their challenges, and evaluated the applicability of states' consolidation lessons to federal challenges.
section_title: What GAO Found
paragraphs: In launching its federal data center consolidation initiative, OMB required the 24 participating agencies to submit data center inventories and consolidation plans by the end of August 2010, and provided guidance on key elements to include in the inventories and plans--such as hardware and software assets, goals, schedules, and cost-benefit calculations. The plans indicate that agencies anticipate closing about 650 data centers by fiscal year 2015 and saving about $700 million in doing so. However, only one of the agencies submitted a complete inventory and no agency submitted complete plans. Further, OMB did not require agencies to document the steps they took, if any, to verify the inventory data. For example, in their inventories, 14 agencies do not provide a complete listing of data centers and 15 do not list all of their software assets. Also, in their consolidation plans, 20 agencies do not reference a master schedule, 12 agencies do not address cost-benefit calculations, and 9 do not address risk management. The reason for these gaps, according to several agency officials, was that they had difficulty completing their inventories and plans within OMB's timelines. Until these inventories and plans are complete, agencies may not be able to implement their consolidation activities and realize expected cost savings. Moreover, without an understanding of the validity of agencies' consolidation data, OMB cannot be assured that agencies are providing a sound baseline for estimating consolidation savings and measuring progress against those goals. Agencies identified multiple challenges during data center consolidation, including those that are specific to OMB's consolidation initiative as well as those that are cultural, funding-related, operational, and technical in nature. For example, in attempting to fulfill OMB's requirements, 19 agencies reported difficulty in obtaining power usage data. In addition, 9 agencies reported challenges in maintaining services during the transition to consolidated services. Moving forward, it will be important for agencies to focus on mitigating such challenges as they implement their consolidation plans. Many state governments have undertaken data center consolidation initiatives in recent years and have encountered challenges similar to those reported by federal agencies. Specifically, 19 states reported lessons learned that could be leveraged at the federal level. For example, a West Virginia official reported that since the state had no funding for data center consolidation, it used the natural aging cycle of hardware to force consolidation; that is, when a piece of hardware was ready to be replaced, the new applications and software were put onto a consolidated server. Also, officials from North Carolina reported that organizations are typically concerned that by consolidating data centers, they will lose control of their data, service levels will decline, or costs will rise. The state learned that during the process of consolidation, the organizations' concerns should be documented, validated, and addressed.
section_title: What GAO Recommends
paragraphs: GAO is recommending that the Federal Chief Information Officer, department secretaries, and agency heads take steps to ensure that agency data center inventories and consolidation plans are complete. Most agencies agreed with GAO's recommendations. Defense and SSA did not agree to complete all missing elements of their inventories and plans. Based on OMB guidance on the importance of these elements, GAO maintains these recommendations to be reasonable and appropriate.

section_title: Letter
paragraphs: The federal government’s demand for information technology (IT) is ever  increasing. In recent years, as federal agencies modernized their  operations, put more of their services online, and increased their  information security profiles, they have demanded more computing power  and data storage resources. Over time, this increasing demand has led to  a dramatic rise in the number of federal data centers and a corresponding  increase in operational costs. The Office of Management and Budget  (OMB) has recognized the significance of this increase and has launched  the Federal Data Center Consolidation Initiative (FDCCI), a  governmentwide effort to consolidate data centers.
This report responds to your request that we review the federal  government’s efforts to consolidate its data centers. Specifically, our  objectives were to (1) assess whether agency consolidation documents  include adequate detail, such as performance measures and milestones,  for agencies to consolidate their centers; (2) identify the key challenges  reported by agencies in consolidating centers; and (3) evaluate whether  lessons learned during state government consolidation efforts could be  leveraged to mitigate challenges at the federal level.
To address our objectives, we reviewed 24 departments’ and agencies’  (agencies) data center inventories and consolidation plans and assessed  their completeness against key elements, as required by OMB. We  interviewed agency officials to determine the extent to which inventory  information had been validated and what challenges to consolidation  federal agencies have faced. Finally, we assessed state reports and  interviewed officials involved with state data center consolidation efforts to  identify lessons learned that could be applied to federal consolidation  efforts.
We conducted this performance audit from August 2010 to July 2011 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives. Appendix I contains further  details about our objectives, scope, and methodology.
subsections: section_title: Background
paragraphs: While the term “data center” can be used to describe any room used for  the purpose of processing or storing data, OMB defines a data center as  a room that is greater than 500 square feet, that is used for processing or  storing data, and that meets stringent availability requirements. Other  facilities are classified as “server rooms,” which are typically less than 500  square feet and “server closets,” which are typically less than 200 square  feet.
According to OMB, the number of federal data centers grew from 432 in  1998 to 2,094 in July 2010. Operating such a large number of centers  places costly demands on the government. While the total annual federal  spending associated with data centers has not yet been determined,  OMB has found that operating data centers is a significant cost to the  federal government, including hardware, software, real estate, and  cooling costs. For example, according to the Environmental Protection  Agency (EPA), the electricity cost to operate federal servers and data  centers across the government is about $450 million annually. According  to the Department of Energy (Energy), data center spaces can consume  100 to 200 times as much electricity as standard office spaces. Reported  server utilization rates as low as 5 percent and limited reuse of these data  centers within or across agencies lends further credence to the need to  restructure federal data center operations to improve efficiency and  reduce costs. In 2010, the Federal Chief Information Officer (CIO)  reported that operating and maintaining such redundant infrastructure  investments was costly, inefficient, and unsustainable.
subsections: section_title: OMB and the Federal CIO Established the Federal Data Center Consolidation Initiative
paragraphs: Concerned about the size of the federal data center inventory and the  potential to improve the efficiency, performance, and environmental  footprint of federal data center activities, in February 2010 OMB, under  the direction of the Federal CIO, announced the Federal Data Center  Consolidation Initiative (FDCCI). This initiative’s four high-level goals are  to    promote the use of “green IT” by reducing the overall energy and real  estate footprint of government data centers; reduce the cost of data center hardware, software, and operations; increase the overall IT security posture of the government; and shift IT investments to more efficient computing platforms and  technologies.
As part of FDCCI, OMB required 24 departments and agencies that  participate on the Chief Information Officers Council (see table 1) to  submit a series of documents that ultimately resulted in a data center  consolidation plan.
Specifically, the departments and agencies were to provide the following:    An initial asset inventory (due April 30, 2010), which was to provide a  high-level understanding of the scale and size of existing data  centers, IT infrastructure assets, and applications supported by the  data centers.   An initial data center consolidation plan (due June 30, 2010), which  was to identify potential areas for consolidation, areas where  optimization through server virtualization or cloud computing  alternatives could be used, and a high-level roadmap for transitioning  to the consolidated end-state architecture.    A final asset inventory baseline (due July 30, 2010), which was to  contain more detailed information and serve as the foundation for  developing the final data center consolidation plans. The final  inventory was also to identify the consolidation approach to be taken  for each data center.   A final data center consolidation plan (due August 30, 2010), which  was to be incorporated into the agency’s fiscal year 2012 budget and  was to include a technical roadmap and approach for achieving the  targets for infrastructure utilization, energy efficiency, and cost  efficiency.
In October 2010, OMB reported that all of the agencies had submitted  their plans and that there were 2,094 federal data centers as of July 2010.  OMB announced plans to monitor agencies’ consolidation activities on an  ongoing basis as part of the annual budget process. Further, starting in  fiscal year 2011, agencies will be required to provide an annual updated  data center asset inventory at the end of every third quarter and a  consolidation progress report at the end of every fourth quarter.
To manage the initiative, OMB designated two agency CIOs as executive  sponsors to lead the effort within the Chief Information Officers Council.  Additionally, the General Services Administration (GSA) has established  the FDCCI Program Management Office, whose role is to support OMB in  the planning, execution, management, and communication for FDCCI. In  this role, GSA collected the responses to the four document deliveries  and reviewed the submissions for completeness and reasonableness.  GSA also sponsored three workshops on the initiative for agencies and  facilitated a peer review of the initial and final data center consolidation  plans.
section_title: OMB’s IT Reform Plan Sets Important Milestones for Data Center Consolidation
paragraphs: In December 2010, OMB published its 25-Point Implementation Plan to  Reform Federal Information Technology Management as a means of  implementing IT reform in the areas of operational efficiency and large- scale IT program management. Among the 25 initiatives, OMB has  included two goals that relate to data center consolidation:  1.  By June 2011, complete detailed implementation plans to consolidate  at least 800 data centers by 2015. 2.  By June 2012, create a governmentwide marketplace for data center  availability.
To accomplish its first goal, OMB required each FDCCI agency to identify  a senior, dedicated data center consolidation program manager. It also  launched a Data Center Consolidation Task Force comprised of the data  center consolidation program managers from each agency. OMB officials  stated that this task force is critical to driving forward on individual agency  consolidation goals and to meeting the overall federal target of closing a  minimum of 800 data centers by 2015. To that end, in April 2011, OMB  announced plans to close 137 data centers by the end of December  2011. OMB also plans to launch a publicly-available dashboard for  observing agencies’ consolidation progress, but this has not yet been  completed.
To accomplish its second goal, OMB and GSA plan to create a  governmentwide marketplace by June 2012 that will better utilize spare  capacity within operational data centers. This online marketplace is  intended to match agencies that have extra capacity with agencies with  increasing demand, thereby improving the utilization of existing facilities.  The marketplace will help agencies with available capacity promote their  available data center space. Once agencies have a clear sense of the  existing capacity landscape, they can make more informed consolidation  decisions.
section_title: GAO Has Previously Reported on Federal Data Center Consolidation Efforts
paragraphs: We have previously reported on OMB’s efforts to consolidate federal data  centers. In March 2011, we reported on the status of the FDCCI and  noted that data center consolidation makes sense economically and as a  way to achieve more efficient IT operations, but that challenges exist. For  example, agencies face challenges in ensuring the accuracy of their  inventories and plans, providing upfront funding for the consolidation  effort before any cost savings accrue, integrating consolidation plans into  agency budget submissions (as required by OMB), establishing and  implementing shared standards (for storage, systems, security, etc.),  overcoming cultural resistance to such major organizational changes, and  maintaining current operations during the transition to consolidated  operations. We further reported that mitigating these and other challenges  will require commitment from the agencies and continued oversight by  OMB and the Federal CIO.
section_title: Agencies Established Consolidation Plans, but Incomplete Inventories and Missing Elements Undermine Expected Savings
paragraphs: To help agencies plan their consolidations, OMB issued guidance on the  required content of data center inventories and consolidation plans.  Specifically, the inventories were to include descriptions of the assets  present within individual data centers, as well as information about the  physical data center itself. The consolidation plans were to address key  elements, including goals, approaches, schedules, cost-benefit  calculations, and risk management plans.
As required, 23 of the 24 agencies submitted their inventories and  consolidation plans by the end of September 2010; the remaining agency  explained that consolidation was not applicable to them. However, of the  23 reporting agencies, all but one of the inventories and all of the plans  are missing key elements. For example, 14 agencies do not provide a  complete listing of data centers and 15 do not provide a complete listing  of software assets in their inventories. Further, OMB did not require that  agencies verify these inventory data. Additionally, in their consolidation  plans, 20 agencies do not provide a master schedule, 12 agencies do not  address cost-benefit calculations, and 9 do not address risk management.  Several agency officials noted that they had difficulty completing their  inventories and plans within OMB’s timelines. Other agencies reported  trouble with identifying either required information for the plans or data on  the assets within their data centers. Until these inventories and plans are  complete, agencies may not be able to implement their consolidation  activities or to realize expected cost savings. Moreover, without an  understanding of the validity of agencies’ consolidation data, OMB cannot  be assured that agencies are providing a sound baseline for estimating  consolidation savings and measuring progress against those goals.
subsections: section_title: Agencies Plan Significant Facility Reductions and Cost Savings
paragraphs: In their consolidation plans, agencies identified goals for reducing the  number of data center facilities and these facilities’ related costs.  Specifically, the 23 reporting agencies identified 1,590 data centers as of  April 2011, and established goals for reducing that number by 652  centers by the end of fiscal year 2015.
Most federal departments and agencies also estimated cost savings over  time. Specifically:    Fourteen agencies reported savings totaling about $700 million  between fiscal years 2011 and 2015; however, actual savings may be  even higher because 12 of these agencies’ estimates were  incomplete. For example, 11 agencies included expected energy  savings and reductions in building operational costs, but not savings  from other sources, such as equipment reductions.   Two agencies expect to accrue net savings after fiscal year 2015.    Two agencies do not expect to attain net savings from their  consolidation efforts.   Five agencies did not provide estimated cost savings; however, two of  these agencies suggested that they plan to develop cost-benefit  analyses in the future.
section_title: Asset Inventories Are Not Complete or Verified
paragraphs: As part of the data center consolidation initiative, OMB required agencies  to provide an inventory of data center assets. This inventory is to address  four key elements: (1) IT software assets; (2) IT hardware assets and  their utilization; (3) IT facilities, energy, and storage; and (4) geographic  location and real estate. According to OMB’s guidance, the information is  to be organized by data center. For example, in the IT software area,  agencies are to report by data center on each major and nonmajor  system present in the center. For each identified system, the agency is to  report the associated support platforms, servers and computers, and  proposed consolidation approach (i.e., decommissioning, consolidation,  cloud computing, or virtualization). Table 2 provides a detailed description  of each of the four key elements.
When collecting data, it is important to have assurance that the data are  accurate. We have previously reported on the need for agencies, when  providing information to OMB, to explain the procedures used to verify  their data. Specifically, agencies should ensure that reported data are  sufficiently complete, accurate, and consistent, and also identify any  significant data limitations. Explaining the limitations of information can  provide a context for understanding and assessing the challenges  agencies face in gathering, processing, and analyzing needed data. Such  a presentation of data limitations can also help identify the actions  needed to improve the agency’s ability to measure its performance. More  recently, we have reiterated the importance of providing OMB with  complete and accurate data and the possible negative impact of that data  being missing or incomplete.
Only 1 of the 23 agency data center inventories contains complete data in  all four of the required elements. Specifically, while many agencies  provide partial inventory data,    one agency provides complete information in all four areas,  five agencies provide complete information in three of the four areas,   one agency provides complete information for two of the areas,    eight agencies have complete information for only one area, and    eight agencies do not have any complete areas in their inventories.
Figure 1 provides an assessment of the completeness of agencies’  inventories, by key element, and a discussion of the analysis of each area  follows the figure.
IT software assets. Eight agencies provide complete information on  their software assets; 14 agencies provide partial information; and 1  agency did not provide information. For example, the Small Business  Administration (SBA) provides information on its data center systems,  their technical dependencies on platforms and servers, and  consolidation approaches for its systems; while GSA provides  information on its data center systems and a consolidation approach  for each system, but provides only partial information on each  system’s technical dependencies on platforms and servers.  Additionally, Energy provides only partial information on the systems   in its data center, the systems’ technical dependencies, and  consolidation approaches for those systems.
IT hardware assets and utilization. Nine agencies provide complete  information on their IT hardware assets and the utilization of those  assets and 14 provide partial information. For example, EPA provides  complete information on maximum and average server utilization, as  well as counts of its physical servers, virtual hosts, and virtual  operating systems; while SBA provides complete information on  counts of its physical servers, virtual hosts, and virtual operating  systems, but only partial information on maximum and average server  utilization. Another 7 agencies, including the Departments of Defense  (Defense), Homeland Security (DHS), and Transportation  (Transportation), provide partial information on their maximum and  average server utilization, and on their counts of physical servers,  virtual hosts, and virtual operating systems.
IT facilities, energy, and storage. Three agencies provide complete  information on their IT facilities, energy, and storage, while 20 provide  partial information. For example, the Department of State (State)  includes all the required information, while the Department of  Education (Education) provides complete information on its annual  data center operational cost, total rack count, and storage information,  but only provides partial information on its annual data center  electricity cost and total electricity usage. Also, the National  Aeronautics and Space Administration (NASA), has partial information  on its total rack count and does not provide any information for the  other required parts of this element.   Geographic location and real estate. Nine agencies provide  complete information on their data center locations, while 14 provide  partial information. For example, the Office of Personnel Management  (OPM) provides complete information on its number of data centers  and the gross floor area of those centers, but does not provide  information on the number of server rooms and closets and the gross  floor area of those facilities. Other agencies such as Energy and the  Department of Labor (Labor) only provide partial information on their  number of centers, server rooms, and closets, and the gross floor  area of those facilities.
Because agency goals are intended to be built on the information  provided by the inventories, agencies cannot ensure the reliability of their  savings and utilization forecasts until the underlying inventories have  been completed.
While it is important that reported data are sufficiently complete, accurate,  and consistent, OMB’s guidance on agency inventories does not require  agencies to document what they did to verify their data, or to disclose any  limitations on that data. Nonetheless, several agencies took informal  steps to validate their data. For example, Department of Agriculture  (Agriculture) officials stated that they interviewed staff who submitted  inventory information and conducted on-site visits of data centers.  Additionally, Department of Commerce (Commerce) officials reported that  they reviewed the inventory data and clarified missing or suspect entries  with those who submitted the information. Also, a Department of the  Treasury (Treasury) official stated that there were two rounds of data  verification and that Treasury bureaus were sometimes asked to verify  submitted information. However, officials from other agencies, such as  Defense, Energy, and NASA, confirmed that their inventories had not  been verified. Further, in some cases, such as with the Department of  Health and Human Services (HHS) and NASA, agency officials reported  that their inventory information was estimated.
Notwithstanding agencies’ informal verification efforts, complete,  accurate, and consistent performance information will be important to  OMB to guide its decisions on how best to oversee federal data center  consolidations. Without an understanding of the validity and limitations on  agencies’ data, OMB cannot be assured that agencies are providing a  sound baseline for estimating savings and measuring progress against  their goals.
section_title: Agencies Established Consolidation Plans, but Work Remains on Key Elements
paragraphs: In addition to the agencies’ inventories, OMB required agencies to  establish consolidation plans that address key elements, including  quantitative goals, qualitative impacts, approach, scope, timeline, and  master schedule, as well as summaries of a cost-benefit analysis,  performance metrics, risk management, and communications planning.  OMB noted the importance of agencies’ consolidation plans in providing a  technical road map and approach for achieving specified targets for  infrastructure utilization, energy efficiency, and cost efficiency. Table 3  provides a detailed description of each of these elements.
While 23 agencies submitted consolidation plans to OMB, selected  elements are missing from each plan. For example, 22 agencies provide  complete information on their qualitative impacts, but only 6 provide  complete information on their quantitative goals. Further, while all 23  agencies specify their consolidation approach, only 5 indicate that a cost- benefit analysis was performed for the consolidation initiative. In many  cases, agencies submitted some, but not all, of the required information.  Figure 2 provides an assessment by element, and a discussion of each  element follows the figure. A detailed summary of the agencies’ status of  completion of each key element is provided in appendix II. In addition, this  information is provided for each agency in appendix III.    Quantitative goals. Six agencies provide complete savings and  utilization forecasts and 17 agencies provide partial forecasts. For  example, Defense’s savings and utilization forecasts are incomplete,  while Treasury and the Social Security Administration (SSA) provide  complete savings forecasts, but incomplete utilization forecasts. Some  agencies identified reasons for not having completed these forecasts.  For example, Treasury’s plan states that the department’s savings  and utilization targets do not include demand from new organizations  required under recent legislation and that the plan will be updated as  further information becomes available. The plan also notes that the  forecasts could change when the department completes associated  cost-benefit analyses. Additionally, NASA’s plan states that the  agency is performing an assessment of its assets to form an accurate  baseline, on which actual targets for reduction can be predicted. This  plan also notes that as the agency’s asset and inventory information is  improved, NASA will evaluate opportunities to further consolidate  applications and virtualize operating systems.   Qualitative impacts. Twenty-two agencies describe the qualitative  impacts of their consolidation initiatives and 1 agency does not. For  example, Agriculture’s plan describes goals such as reducing overall  energy use and reducing the real estate footprint for data centers.  Additionally, HHS reports that the consolidation effort will result in  more efficient monitoring of data center power. Finally, the U.S.  Agency for International Development (USAID) describes goals such  as optimizing use of IT funding by increasing efficiencies and  increasing the availability of resources and systems to the user  community. However, Education does not provide any qualitative  impacts. Although the department links its consolidation plan to a  federal strategic sustainability plan, the sustainability plan does not  contain qualitative impacts such as those required by OMB.   Summary of consolidation approach. All 23 agencies include a  summary of the agencies’ proposed consolidation approaches. For  example, Commerce describes five approaches that will support the  department’s FDCCI goals: consolidating and decommissioning data  centers, increasing server virtualization and IT equipment utilization,  moving to cloud computing, acquiring green products and services,  and promoting “green IT.” Similarly, the Department of the Interior  (Interior) provides four approaches to help realize its consolidation  goals: decommissioning, consolidation, cloud computing, and server  and storage virtualization. For a more detailed discussion of  alternative data center consolidation approaches, see appendix IV.   Scope of consolidation. Nineteen agencies’ plans include a well- defined scope for data center consolidation, 2 provide partial  information on the scope of their consolidation efforts, and 2 do not  provide this information. Specifically, the agencies that provide this  information list the data centers included in the consolidation effort  and what consolidation approach will be taken for the systems within  each center (i.e., decommissioning, consolidation, cloud computing,  or virtualization). For example, the Department of Veterans Affairs  (VA) lists 87 data centers and Labor lists 20 data centers, in which all  of the systems will be consolidated, decommissioned, or virtualized.  Alternatively, Energy has not yet determined what action will be taken  for each of its facilities and Interior identifies the total numbers of  centers to be retained or expanded, but does not describe the  consolidation approach for individual data centers. Two agencies, the  Department of Justice (Justice) and NASA, are still working to  determine which of their data centers are to be consolidated.   High-level timeline. Twenty agencies include a high-level timeline for  consolidation efforts, 1 agency includes partial information on its  timelines, and 2 do not provide timelines. For example, Labor and VA  both provide the fiscal years in which every data center listed will be  consolidated and the National Science Foundation (NSF) states what  year the agency’s primary data center will be decommissioned and  replaced with private and public cloud services. In contrast, Defense  only describes broad goals to be accomplished by fiscal year 2013  and does not include specific milestones for each data center.  Further, NASA does not include this information in its plan and notes  that the agency is still working to determine its data center inventory.   Performance metrics. Six agencies identify specific performance  metrics for their consolidation programs, 4 agencies provide partial  information on their metrics, and 13 agencies did not identify specific  metrics. For example, both Transportation and GSA specify metrics  such as savings in energy consumption, cost variance, and schedule  variance. Alternatively, State reports that the department’s data center  consolidation program maintains metrics at both the system and  process performance levels, but does not provide any specifics as to  the nature of those metrics. Further, although Defense does not  provide metrics at the department level, the Air Force has developed  a method to provide such measures.   Master program schedule. Three agencies reference a completed  master program schedule, and 20 do not. For example, while  Agriculture, DHS, and Interior discuss their master schedules, other  agencies, such as Commerce and HHS do not. Some agencies, such   as Defense, Labor, and the Nuclear Regulatory Commission (NRC),  plan to develop them in the future.   Cost-benefit analysis. Five agencies provide a cost-benefit analysis  that encompasses their entire consolidation initiative, 6 agencies  provide only selected elements of a cost-benefit analysis, and 12  agencies do not provide a cost-benefit analysis. For example, DHS  details full annualized investment and savings estimates through fiscal  year 2015, while other agencies, such as State and OPM, provide  only partial information. Specifically, State acknowledges that not all  costs are accounted for in its analysis and OPM reports that its  analysis is preliminary. Additionally, Commerce provides costs and  savings for several data center consolidations but acknowledges that  estimates cannot be provided for all of the department’s planned  consolidation initiatives. Eight of the agencies that do not provide a  cost-benefit analysis, such as HHS, Justice, and USAID, plan to  conduct one in the future.   Risk management. Fewer than half of the agencies both reference a  consolidation risk management plan and require that risks be tracked.  For example, HHS discusses its approach to risk management and  identifies a series of technical, security, funding, and management  risks and provides a mitigation strategy for each. Additionally, VA  describes a five-phase approach to risk management that includes  identifying and monitoring risks. However, Education requires that  risks be tracked, but does not reference the existence of an actual risk  management plan. Nine agencies do not reference a risk  management plan or requirements for tracking risks.   Communications plan. Eighteen agencies consider a  communications strategy for the agencies’ consolidation initiatives,  and 5 agencies do not. For example, Energy describes a series of  coordinated activities that are intended to support the consolidation  effort. Additionally, NASA details its approach to consolidation  coordination and communication and SBA details individual  communication responsibilities among consolidation stakeholders.  However, Treasury and NRC do not describe such a communications  strategy.
When asked about the elements missing from their plans, many agency  officials stated that they completed what they could within the timelines  provided by OMB. Several agency officials noted that it was difficult to  obtain all of the required data from component agencies, while others  reported that their data collection efforts were made more difficult by  OMB’s tight time frames and changes in templates and guidance.  Moreover, officials from two agencies stated that some of the information  contained in their plans had been estimated. However, OMB has not  required agencies to complete the missing elements or to resubmit their  final plans. According to an OMB official, agencies have been instructed  to move forward with their consolidation initiatives, and as noted earlier,  OMB intends to monitor the agencies’ progress annually.
We have previously reported that without a clear description of the  strategies and resources an agency plans to use in meeting its goals, it  will be difficult to assess the likelihood of the agency’s success in  achieving its intended results. In the absence of completed  consolidation plans, agencies run the risk of moving forward on their  respective initiatives with, among other things, poorly defined approaches  and outcomes. Without this information, agencies may not realize  anticipated cost savings, improved infrastructure utilization, and energy  efficiency.
section_title: Agencies Face Multiple Challenges in Consolidation
paragraphs: In preparing agencies for the data center consolidation initiative, OMB  held workshops that, among other things, discussed challenges that  agencies might face so that they could anticipate and mitigate them. In  addition, agencies identified multiple challenges they are facing during  data center consolidations. These include challenges related to the data  center consolidation initiative as well as those that are cultural, funding  related, operational, and technical. Some challenges are more common  than others. Specifically, the number of agencies reporting a particular  challenge range from 1 agency to 19 agencies. Table 4 details the  reported challenges as well as the numbers of agencies experiencing that  challenge. The table is followed by a discussion of the most prevalent  challenges.
Agencies reported nine challenges that are specific to OMB’s data center  consolidation initiative, including meeting tight FDCCI deadlines and  obtaining power usage data as required by OMB. Specifically, 19  agencies reported that obtaining power usage data was a challenge. For  example, a Commerce official stated that while more than half of the  agency’s data centers have power consumption figures or costs  associated with power usage, some of the agency’s facilities do not have  metering capabilities for power consumption and that the agency has  some server rooms that lack metering at the equipment. Similarly, Labor’s  Deputy CIO stated that the department does not have metering for data  centers. Consequently, the agency used best practices to estimate how  much power is being used at data centers. In addition, 11 agencies found  that the tight FDCCI deadlines were a challenge. For example, Labor’s  Deputy CIO stated that the time frames were overly aggressive and did  not allow the agency to provide the information OMB requested or to  complete the planning that is necessary for such an important  undertaking. An Energy official stated that there is no quick path to  consolidation and that the agency was faced with the decision of either  moving forward with inadequate inventory information or taking more time  to make decisions. The official stressed that the agency would like to  consolidate in the correct manner.
Agencies reported four cultural challenges to data center consolidation,  including accepting cultural change and implementing consolidation in an  organizational structure not geared towards consolidation (i.e., a  decentralized enterprise). The most prevalent challenge was acceptance  of cultural change, with 15 agencies reporting it as a challenge. For  example, an Agriculture official stated that there is a challenge in  addressing cultural change surrounding data center consolidation. With  data center consolidation, systems personnel may not be in the same  location as the data. To address this, Agriculture refined its  communications plan so that lessons learned can be passed on to other  staff. An EPA official noted the agency experienced this challenge when  employees were reluctant to cede control of resources under their  immediate control. EPA mitigated that challenge by building relationships  with stakeholders early on. In addition, 8 agencies reported that  implementing the consolidation was challenged by their organizational  structures. For example, a Justice official stated that the agency uses a  federated IT approach within which the departmentwide CIO’s office has  primary responsibility for architecture, common infrastructure, and  standards decisions, while each component IT department has primary  responsibility for application resource decisions. According to this official,  such a federated approach offers Justice’s components more autonomy  when making decisions, but creates obstacles for departmentwide efforts  such as FDCCI. Further, an Energy official cited the department’s  decentralized environment as a challenge in being able to collect data  center inventory information.
The data center consolidation initiative is supposed to result in cost  savings, but multiple agencies reported challenges in funding their  initiatives. For example, a Transportation official reported that the need  for upfront funding for consolidation efforts was a challenge. A NASA  official stated that the agency spent approximately $1.5 million on an  asset management tool to assist the agency in creating its inventory.  Further, a State official noted the challenge of having to fund the  consolidation efforts long before cost savings will be realized. In addition,  9 agencies reported that identifying cost savings for consolidation efforts  was a challenge. For example, an Energy official stated that it was too  early in the consolidation process for Energy to be able to quantify cost  savings, since the agency does not have data on cost or exactly which  data centers will be closed. A State official stated that it is difficult to  estimate cost savings since the department does not have information on  power usage for all facilities. Further, a Treasury official noted challenges  in identifying cost savings, particularly because a reduction in utilized  square footage at a facility does not mean that the leasing agency will  issue a refund check on the lease. Similarly, building managers at private  facilities will typically not issue a refund if an agency begins using less  energy.
Agencies reported eight operational challenges to data center  consolidation, including maintaining services during the consolidation  transition and implementing cloud computing. Nine agencies reported that  maintaining services during the consolidation transition is a challenge. For  example, a Labor official stated that keeping the business running during  the transition is a big concern of the agency. Three agencies reported that  moving to cloud computing was a challenge. For example, a Commerce  official stated that the agency’s biggest challenge was how to implement  cloud computing. This official cited the need to investigate how private  and public cloud computing will fit into the agency’s mission, and to  determine how to manage security issues surrounding cloud computing.
Agencies reported 10 technical challenges to data center consolidation,  including maintaining the appropriate level of system security and  planning the migration strategy. While agencies reported more technical  challenges than any other type of challenge, these challenges are more  diverse, with fewer agencies experiencing each individual challenge.  Three agencies reported that maintaining the appropriate level of system  security was a challenge. For example, an OPM official stated that one of  the challenges faced was the need to maintain personally identifiable  information while exploring options such as cloud computing. In addition,  a State official identified the challenge of including classified servers in  the consolidation initiative. Two agencies reported that planning the  migration strategy was a challenge. For example, an SSA official pointed  out the difficulty in scheduling migration across approximately 1,500 field  offices.
One approach agencies can use to manage challenges such as the ones  listed above is through formal risk management processes. However, as  noted in the prior section, less than half of the agencies included a  discussion of risk management in their data center consolidation plans.
section_title: Lessons Learned by States Can Be Leveraged to Mitigate Challenges at the Federal Level
paragraphs: We have previously reported on the importance of using lessons  learned—a principal component of an organizational culture committed to  continuous improvement. Sharing such information serves to  communicate acquired knowledge more effectively and to ensure that  beneficial information is factored into planning, work processes, and  activities. Lessons learned also provide a powerful method of sharing  good ideas for improving work processes, facility or equipment design  and operation, quality, safety, and cost-effectiveness.
OMB posted lessons learned by three states on its data center  consolidation Web page for federal agencies to review. However, there  is more that agencies can learn. Many state governments have  undertaken data center consolidation initiatives in recent years. Although  they have encountered unique challenges, they have also encountered  challenges similar to those reported by federal agencies. Specifically, the  National Association of State CIOs and a literature search identified 20  states that reported on challenges they faced, or lessons they learned,  from their data center consolidation initiatives. Of these, 19 reported  lessons learned that could be leveraged at the federal level. For example,  officials from North Carolina reported that organizations are typically  concerned that by consolidating data centers, they will lose control of their  data, service levels will decline, or costs will rise. The state learned that to  help mitigate this during the process of consolidation, the organizations’  concerns should be documented, validated, and addressed. In another  example, a West Virginia official reported that since the state had no  funding for a consolidation, it had to be creative in executing the  consolidation. The state used the natural aging cycle of hardware to force  consolidation; that is, when a piece of hardware was ready to be  replaced, the applications and software were put onto a consolidated  server. As a final example, two states reported lessons learned that could  be applied to the challenge of providing a quality asset inventory. Officials  from Utah and Texas emphasized the importance of having an accurate  inventory of all equipment that could be impacted by the project. The  official from Texas added that it is important to have a third party collect  technical data across agencies, and that it is beneficial to have this work  completed by outside parties, in order to ensure objectivity and  consistency. Table 5 identifies lessons learned by states that could be  applied by federal agencies.
section_title: Conclusions
paragraphs: With agencies reporting having almost 1,600 federal data centers, OMB’s  goal of consolidating 800 centers by 2015 is ambitious. To its credit, OMB  has established an accountability infrastructure through its data center  consolidation task force composed of representatives from each of the  participating agencies. OMB and federal agencies have also taken  important steps to reduce the number and increase the efficiency of the  federal data centers. However, only one agency has completed its  required data center asset inventory, no agencies have completed their  consolidation plans, and OMB has not required that agency inventory  information be verified. Despite these limitations, OMB has instructed  agencies to move forward with their plans. Moving forward to consolidate  obviously redundant or underutilized centers is warranted—and should  result in immediate cost savings and increased efficiency. However,  without a complete asset inventory and a comprehensive plan, agencies  are at increased risk that they will be ill-prepared to manage such a  significant transformation. This could slow the consolidations and reduce  expected savings and efficiencies.
In moving ahead in their consolidation efforts, agencies are encountering  challenges, including those that are technical, operational, and cultural in  nature. Some state governments have also engaged in data center  consolidation initiatives and dealt with similar obstacles in doing so. By  virtue of these experiences, these states can offer insights and  suggestions that federal agencies can use to mitigate their challenges  and risks. In doing so, agencies will be better positioned to address their  consolidation goals and to meet OMB’s goals for reducing the number  and cost of federal data centers.
section_title: Recommendations for Executive Action
paragraphs: To better ensure that the federal data center consolidation initiative  improves governmental efficiency and achieves cost savings, we are  making four recommendations to OMB. Specifically, we are  recommending that the Director of the Office of Management and Budget  direct the Federal Chief Information Officer to  require that agencies, when updating their data center inventories in  the third quarter of each fiscal year, state what actions have been  taken to verify the inventories and to identify any limitations of this  information; require that agencies complete the missing elements in their  respective plans and submit complete data center consolidation plans,  or provide a schedule for when they will do so, by September 30,  2011; require agencies to consider consolidation challenges and lessons  learned when updating their plans; and   utilize the existing accountability infrastructure by requiring the Data  Center Consolidation Task Force to assess agency consolidation  plans to ensure they are complete and to monitor the agencies’  implementation of their plans.
In addition, we are making two recommendations to each of the  department secretaries and agency heads of the 23 departments and  agencies participating in the federal data center consolidation initiative.  Specifically, we are recommending that the secretaries and agency heads    direct their component agencies and their data center consolidation  program managers to complete the missing elements in their  respective data center consolidation inventories and plans; and require their data center consolidation program managers to consider  consolidation challenges and lessons learned when updating their  consolidation plans.
section_title: Agency Comments and Our Evaluation
paragraphs: We received comments on a draft of our report from OMB and the 23  agencies to which we made recommendations. Most agencies generally  agreed with our recommendations. Specifically, in commenting on the  draft, 15 agencies agreed with our recommendations; 4 agreed with the  report’s content or findings, but offered no comments on the  recommendations; 3 offered no comments on the report’s findings or  recommendations; and Defense and SSA both did not agree with one of  our recommendations, but agreed with the second. Agencies also  provided technical comments, which we incorporated as appropriate.  Each agency’s comments are discussed in more detail below.
In comments provided via e-mail, an OMB official from the General  Counsel Office wrote that the agency generally agreed with our report.  The agency offered no comments on our recommendations.
In comments provided via e-mail, Agriculture’s CIO agreed with our  recommendations and noted that our assessment of USDA’s  inventory was accurate.
In written comments, the Secretary of Commerce concurred with the  general findings as they apply to the department and with specific  reporting on the department’s data center consolidation plan. The  Secretary offered no comments on our recommendations, but noted  that Commerce plans to address GAO’s finding on the department’s  consolidation master program schedule in the next version of its  consolidation plan. Commerce’s written comments are provided in  appendix V.
In written comments, Defense’s CIO partially concurred with one of  our recommendations and concurred with the second. Specifically,  regarding our recommendation that the department complete the  missing elements from its data center inventory and consolidation  plan, the CIO cited the importance of completing consolidation metrics  and noted that many of the department’s centers and buildings are not  equipped to meter energy usage and that using incomplete estimates  of such usage would result in inaccurate extrapolations of cost  savings. However, OMB addressed such concerns in its guidance on  the FDCCI, noting alternative means by which agencies could develop  energy utilization estimates. OMB further recognized that these  estimates may need time to become more accurate. As such, we  believe our recommendation is reasonable and appropriate.  Defense’s written comments are provided in appendix VI.
In written comments, Education’s CIO concurred with one  recommendation and outlined plans to address the second. The CIO  noted the department’s plans to complete, to the extent practicable,  the missing information in Education’s data center inventory and  consolidation plan. The CIO also cited the department’s intent, when  updating its annual consolidation plan, to consider relevant  consolidation challenges and lessons learned. Education’s written  comments are provided in appendix VII.
In written comments, Energy’s Director of the Corporate IT Project  Management Office agreed with our assessment of Energy’s data  center consolidation plan and offered no comments on our  recommendations. The Director cited a series of planned actions by  the department intended to gather missing information in Energy’s  data center inventory, update the department’s consolidation plan,  and document its data center management best practices. Energy’s  written comments are provided in appendix VIII.
In written comments, HHS’ Assistant Secretary for Legislation stated  that the draft accurately depicts the HHS data center consolidation  plan as it was delivered to OMB in August 2010 and notes that the  agency has continued to improve its inventory and make progress on  its data center consolidation goals since that time. Further, the  Assistant Secretary outlined a series of actions planned by the  department to complete HHS’ data center inventory and consolidation  plan. The department did not offer comments on our  recommendations. HHS’ written comments are provided in appendix  IX.
In written comments, DHS’s Director of the Departmental GAO/OIG  Liaison Office concurred with our recommendations. Further, the  Director outlined the department’s planned actions to complete the  missing information from its data center inventory and noted that the  department is working to share its consolidation lessons learned with,  among others, the Federal Chief Information Officers Council. DHS’s  written comments are provided in appendix X.
In written comments, Interior’s Assistant Secretary for Policy,  Management, and Budget concurred with our findings and  recommendations. The Assistant Secretary also noted that the  department is continuing to refine its data center inventory and  consolidation goals. Interior’s written comments are provided in  appendix XI.
In comments provided via e-mail, the Justice Audit Liaison concurred  with our recommendations. Justice also provided technical comments,  which we incorporated as appropriate.
In written comments, Labor’s Assistant Secretary for Administration  and Management stated that, after carefully reviewing the draft report,  the department did not have any comments to contribute. Labor’s  written comments are provided in appendix XII.
In written comments, State’s Chief Financial Officer concurred with  our recommendations and outlined a series of actions planned by the  department to complete State’s data center inventory and  consolidation plan. State’s written comments are provided in appendix  XIII.
In comments provided via e-mail, Transportation’s Deputy Director of  Audit Relations stated that the department had no comments on the  report and agreed to consider our recommendations.
In written comments, Treasury’s Deputy Assistant Secretary for  Information Systems and Chief Information Officer did not provide  comments on our recommendations, but noted that Treasury has  started its annual data center inventory collection, which will address  missing data elements, and that Treasury intends to collect and  leverage data center consolidation challenges and lessons learned  when updating the department’s consolidation plans. Treasury’s  written comments are provided in appendix XIV.
In written comments, VA’s Chief of Staff generally agreed with the  findings and concurred with our recommendations. Further, the Chief  of Staff noted planned actions to complete missing information from  the department’s consolidation plan and to supplement updates to the  plan with narrative responses on consolidation challenges and  lessons learned. The Chief of Staff also noted that the department is  continuing to refine its data center inventory, consolidation goals, and  consolidation timeline. VA’s written comments are provided in  appendix XV.
In written comments, EPA’s Assistant Administrator and Chief  Information Officer did not agree or disagree with our  recommendations, but did offer clarification on its plans to fulfill our  recommendations. Specifically, in relation to our recommendation to  complete missing elements of the agency’s consolidation plan, the  Assistant Administrator clarified that the majority of EPA server rooms  are located within leased space managed by GSA. As such, EPA will  be able to estimate operating electrical use for these rooms, but the  cost-benefit analysis will not reflect a reduction in real estate costs or  electricity consumption because such reductions would not result in  cost savings to EPA.
Further, the CIO asserted that we had mischaracterized the agency’s  overall plan for consolidation by stating that EPA does not plan to  further consolidate its four data centers and in our description of  actions the agency plans to take within those four centers.  Specifically, EPA stated that we did not provide adequate detail about  EPA’s existing infrastructure or the services that will be provided by  these four existing centers. However, EPA’s data center consolidation  plan states that the agency had four data centers as of the end of  fiscal year 2010 and the agency plans to have four data centers at the  end of fiscal year 2014. As such, we maintain that our description of  EPA’s broad consolidation goals is factual. EPA’s written comments  are provided in appendix XVI.
In written comments, GSA’s Administrator agreed with both our  findings and our recommendations and stated that GSA would take  actions commensurate with our recommendations. GSA’s written  comments are provided in appendix XVII.
In written comments, NASA’s CIO concurred with our  recommendations. Further, the CIO cited ongoing work by NASA to  complete the missing information in the agency’s data center  inventory and consolidation plan. Additionally, the CIO noted the  agency’s plans to consider consolidation challenges and lessons  learned when updating consolidation plans. NASA’s written comments  are provided in appendix XVIII.
In comments provided via e-mail, NSF’s Acting CIO did not provide  comments on our recommendations, but noted NSF’s planned actions  to complete the missing information from the agency’s consolidation  plan.
In written comments, NRC’s Deputy Executive Director for Corporate  Management within the Office of the Executive Director for Operations  stated that the agency had no comments. NRC’s written comments  are provided in appendix XIX.
In comments provided via e-mail, OPM’s Deputy CIO concurred with  our recommendations. The Deputy CIO noted that since OPM does  not plan to consolidate further than its one data center, the agency’s  consolidation focus will be to complete its asset inventory and explore  ways to operate more efficiently.
In comments provided via e-mail, the SBA Program Manager for the  Office of Congressional and Legislative Affairs concurred with our  recommendations.
In written comments, SSA’s Deputy Chief of Staff disagreed with one  recommendation and agreed with the second. Specifically, regarding  our recommendation that the agency complete the missing elements  from its data center inventory and consolidation plan, the Deputy  Chief of Staff disagreed with our assessment of SSA’s asset inventory  and consolidation plan, stating that SSA responded to OMB’s directive  in a timely and satisfactory manner. The Deputy Chief of Staff further  noted that because SSA does not plan to consolidate its two existing  data centers, the plan elements we noted as missing were not  applicable to SSA’s circumstances.
However, in its asset inventory, SSA does not provide one of the  OMB-specified consolidation approaches, which includes an option of  “not applicable,” for any of the major and nonmajor systems in SSA’s  data centers. While we acknowledge that SSA does not plan to  consolidate from its two physical locations, OMB still required  agencies to provide a consolidation approach for every identified  system. Further, in a written response to OMB questions about SSA’s  consolidation plan, SSA acknowledged that the agency planned to  virtualize systems within one of its two locations. In light of this  planned work, it is reasonable to assume that an agency would  complete the important governance-related key plan elements we  identified as missing, such as a master program schedule, a cost- benefit analysis, and a risk management plan. In its guidance on the  FDCCI, OMB echoed this importance, noting that an agency’s  governance framework needs to provide specific details about the  oversight and internal mechanics that will measure and manage  performance and risk of the consolidation implementation. As such,  we believe our recommendation is reasonable and appropriate. SSA’s  written comments are provided in appendix XX.
In comments provided via e-mail, the liaison from USAID’s Office of  the Chief Financial Officer concurred with our recommendations.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to interested congressional  committees; the Director of OMB; the secretaries and agency heads of  the departments and agencies addressed in this report; and other  interested parties. In addition, the report will be available at no charge on  the GAO Web site at http://www.gao.gov.
If you or your staff have any questions on the matters discussed in this  report, please contact me at (202) 512-6253 or willemssenj@gao.gov.  Contact points for our Offices of Congressional Relations and Public  Affairs may be found on the last page of this report. GAO staff who made  major contributions to this report are listed in appendix XXI.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to (1) assess whether agency consolidation  documents include adequate detail, such as performance measures and  milestones, for agencies to consolidate their centers; (2) identify the key  challenges reported by agencies in consolidating centers; and (3)  evaluate whether lessons learned during state government consolidation  efforts could be leveraged to mitigate challenges at the federal level.
For this governmentwide review, we assessed the 24 departments and  agencies (agencies) that were identified by the Office of Management and  Budget (OMB) and the Federal Chief Information Officer (CIO) to be  included in the Federal Data Center Consolidation Initiative (FDCCI).  Table 6 lists these agencies.
To evaluate the agencies’ data center inventories and consolidation  plans, we reviewed OMB’s guidance and identified key required elements  for each. We compared agency consolidation inventories and plans to  OMB’s required elements, and identified gaps and missing elements. We  rated each element as “Yes” if the agency provides complete information;  “Partial” if the agency provides some, but not all, of the information; and  “No” if the agency did not provide the information. We followed up with  agencies to clarify our initial findings and to determine why parts of the  inventories and plans were incomplete or missing. We assessed the  reliability of the data agencies provided in their data center inventories  and plans. Specifically, we interviewed agency officials to determine how  the data in the inventories and plans had been collected and their  processes for ensuring the reliability of the data contained in these  inventories. We reviewed the inventories and plans for omissions,  outliers, and typographic mistakes. We compared inventory summary  data contained in the consolidation plans to inventories and noted any  inconsistencies. In doing so, we found multiple gaps in agency-provided  data. We also found that almost half of the agencies had not taken steps  to verify their inventory data. We have reported on these limitations in the  body of this report.
To identify the key challenges encountered by agencies in consolidating  data centers, we analyzed available literature on data center  consolidation challenges and interviewed agency officials to determine  what challenges to consolidation had been encountered. We then  categorized the agency-reported challenges to determine ones that were  encountered most often.
To evaluate whether lessons learned during state government  consolidation efforts could be leveraged to mitigate challenges at the  federal level, we conducted a literature search for information on state  experiences in data center consolidation and interviewed the National  Association of State Chief Information Officers regarding states’  experiences with data center consolidation. These sources identified 20  states that reported challenges or lessons learned from their data center  consolidation initiatives. We sought clarification on challenges and  lessons learned through e-mail and interviews with state officials. We  compared states’ challenges and lessons learned to the challenges facing  federal agencies in order to identify which lessons learned could be  applied to federal consolidation efforts.
We conducted our work at multiple agencies’ headquarters in the  Washington, D.C., metropolitan area. We conducted this performance  audit from August 2010 to July 2011, in accordance with generally  accepted government auditing standards. Those standards require that  we plan and perform the audit to obtain sufficient, appropriate evidence to  provide a reasonable basis for our findings and conclusions based on our  audit objectives. We believe that the evidence obtained provides a  reasonable basis for our findings and conclusions based on our audit  objectives.
section_title: Appendix II: Assessment of Agencies’ Completion of Consolidation Planning Elements, Arranged by Key Element
paragraphs: Table 7 identifies the agencies that provide complete, partial, or no  information for each key element of its data center consolidation plan.
section_title: Appendix III: Assessment of Agencies’ Completion of Key Consolidation Planning Elements, Arranged by Agency
paragraphs: As part of its data center consolidation initiative, OMB required 24 federal  departments and agencies to submit a data center inventory and a data  center consolidation plan. Key elements of the inventory were to include,  for each data center, information on IT hardware, IT software,  facilities/energy/storage, and geographic location. Key elements of the  plan were to include information on quantitative goals, qualitative impacts,  consolidation approach, consolidation scope, timeline, performance  metrics, master schedule, cost-benefit analysis, risk management, and  consideration of a communications plan.
For each of the agencies, the following sections provide a brief summary  of the agencies’ goal for reducing the number of data centers, and an  assessment of the completeness of their inventories and plans.
The following information describes the key that we used in tables 8  through 30 to convey the results of our assessment of the agencies’  compliance with OMB’s requirements for the FDCCI. ●  The agency provides complete information for this element.   ◐  The agency provides some, but not all, aspects of the element.  ○  The agency does not provide information for this element.
subsections: section_title: Department of Agriculture
paragraphs: Agriculture plans to consolidate from 46 data centers to 7 by fiscal year  2015. However, the agency’s asset inventory and consolidation plan are  not complete. In its asset inventory, the department provides complete  information for 1 element and partial information for the remaining 3  elements. Additionally, in its consolidation plan, Agriculture provides  complete information for 8 of the 10 elements, partial information for 1  element, and does not provide information for the remaining element.  Table 8 provides our assessment of Agriculture’s compliance with OMB’s  requirements.
Commerce plans to consolidate from 41 data centers to 23 by fiscal year  2015. However, the agency’s asset inventory and consolidation plan are  not complete. In its inventory, the department provides complete  information for 3 of the 4 key elements and partial information for the  remaining element. Additionally, in its consolidation plan, Commerce  provides complete information for 8 of the 10 elements evaluated,  provides partial information for 1 element, and does not provide  information for the remaining element. Table 9 provides our assessment  of Commerce’s compliance with OMB’s requirements.
Defense plans to consolidate from 772 data centers to 532 by fiscal year  2013. However, the agency’s asset inventory and consolidation plan are  not complete. In its asset inventory, Defense provides only partial  information for all 4 key elements. Additionally, in its consolidation plan,  Defense provides complete information for 3 of the 10 elements  evaluated, provides partial information for 5 elements, and does not  provide information for the remaining 2 elements. A Defense official  explained that because they are a decentralized agency and are fighting  multiple wars, it was difficult to meet OMB’s extremely short deadlines.  The official also noted that OMB’s changing templates and definitions  made it more difficult to compile the needed information. Table 10  provides our assessment of Defense’s compliance with OMB’s  requirements.
Education does not have plans to consolidate any of its three data  centers before fiscal year 2015. Rather, the agency plans to increase  server virtualization within in its centers. However, Education’s asset  inventory and consolidation plan are not complete. In its asset inventory,  the agency provides complete information for 3 key elements and  provides partial information for the remaining element. Additionally, in its  consolidation plan, Education provides complete information for 4 of the  10 elements evaluated, provides partial information for 3 elements, and  does not provide information for the remaining 3 elements. Education  officials stated that they did not provide selected plan elements because  they are not applicable given the agency’s focus on virtualization rather  than consolidation. Table 11 provides our assessment of Education’s  compliance with OMB’s requirements.
Energy plans to consolidate from 31 data centers to 25 by fiscal year  2015. However, the agency’s asset inventory and consolidation plan are  not complete. In its asset inventory, Energy reports that although it began  identifying contractor-operated data centers, this initiative was not  completed in time to be included in the inventory. Additionally, in its  consolidation plan, Energy provides complete information for 3 of the 10  elements evaluated, provides partial information for 2 elements, and does  not provide information for the remaining 5 elements. Table 12 provides  our assessment of Energy’s compliance with OMB’s requirements.
HHS plans to consolidate from 185 data centers to 131 by fiscal year  2015. However, the agency’s asset inventory and consolidation plan are  not complete. In its asset inventory, the department provides partial  information for all 4 key elements. Additionally, in its consolidation plan,  HHS provides complete information for 6 of the 10 elements evaluated,  provides partial information for 1 element, and does not provide  information for the remaining 3 elements. Agency officials stated that they  are working to complete the elements that are missing or incomplete.  Table 13 provides our assessment of HHS’ compliance with OMB’s  requirements.
DHS plans to consolidate from 43 data centers to 2 by fiscal year 2014.  However, the agency’s asset inventory and consolidation plan are not  complete. In its asset inventory, DHS provides partial information for all 4  key elements. Additionally, in its consolidation plan, the department  provides complete information for 8 of the 10 elements evaluated,  provides partial information for 1 element, and does not provide  information for the remaining element. Table 14 provides our assessment  of DHS’s compliance with OMB’s requirements.
The Department of Housing and Urban Development did not submit a  data center inventory or consolidation plan. Instead, it submitted a letter  that asserts that the department does not own any data centers and has  no arrangements to take ownership of any data centers at the end of any  contracts.
section_title: Department of the Interior
paragraphs: Interior plans to consolidate from 95 data centers to 5 by fiscal year  2015. However, the agency’s asset inventory and consolidation plan are  not complete. In its asset inventory, Interior provides complete information  for 1 key element and partial information for the remaining 3 elements. In  its consolidation plan, Interior provides complete information for 8 of the  10 elements evaluated, and provides partial information for 2 elements.  Interior officials stated that they are working to complete the elements that  are missing or incomplete. Table 15 provides our assessment of Interior’s  compliance with OMB’s requirements.
Justice plans to consolidate from 65 data centers to 50 by fiscal year  2015. However, the agency’s asset inventory and consolidation plan are  not complete. In its asset inventory, Justice provides complete information  for 1 key element and partial information for the remaining 3 elements.  Additionally, in its consolidation plan, Justice provides complete  information for 5 of the 10 elements evaluated, provides partial  information for 2 elements, and does not provide information for the  remaining 3 elements. Table 16 provides our assessment of Justice’s  compliance with OMB’s requirements.
Labor plans to consolidate from 20 data centers to 18 by fiscal year 2015.  However, the agency’s asset inventory and consolidation plan are not  complete. In its asset inventory, Labor provides complete information for  1 key element and provides partial information for the remaining 3  elements. Additionally, in its consolidation plan, Labor provides complete  information for 7 of the 10 elements evaluated, provides partial  information for 1 element, and does not provide information for the  remaining 2 elements. Table 17 provides our assessment of Labor’s  compliance with OMB’s requirements.
State plans to consolidate from 13 data centers to 6 by fiscal year 2015.  However, the agency’s asset inventory and consolidation plan are not  complete. In its asset inventory, State provides complete information for 3  of the key elements and provides partial information for the remaining  element. Additionally, in its consolidation plan, State provides complete  information for 6 of the 10 elements evaluated, provides partial  information for 3 elements, and does not provide information for the  remaining element. An agency official stated that they have a master  program schedule and performance metrics, but acknowledged that they  did not provide them to OMB as part of their consolidation plans. Table 18  provides our assessment of State’s compliance with OMB’s requirements.
Transportation plans to consolidate from 35 data centers to 31 by fiscal  year 2015. However, the agency’s asset inventory and consolidation plan  are not complete. In its inventory, Transportation provides partial  information for all 4 key elements, noting that in some instances, data  center owners did not provide the requested information. Additionally, in  its consolidation plan, Transportation provides complete information for 9  of the 10 elements evaluated and does not provide information for the  remaining element. Table 19 provides our assessment of Transportation’s  compliance with OMB’s requirements.
Treasury plans to consolidate from 42 data centers to 29 by fiscal year  2015. However, the agency’s asset inventory and consolidation plan are  not complete. In its inventory, Treasury provides complete information for  1 key element and provides partial information for the remaining 3  elements. Additionally, in its consolidation plan, Treasury provides  complete information for 5 of the 10 elements evaluated, provides partial  information for 1 element, and does not provide information for the  remaining 4 elements. An agency official stated that the agency is  working to complete the missing or incomplete items. Table 20 provides  our assessment of Treasury’s compliance with OMB’s requirements.
VA plans to consolidate 87 data centers into 4 by fiscal year 2015.  However, the agency’s asset inventory and consolidation plan are not  complete. In its inventory, VA provides partial information for all 4 key  elements. Additionally, in its consolidation plan, VA provides complete  information for 6 of the 10 elements evaluated, provides partial  information for 2 elements, and does not provide information for the  remaining 2 elements. Table 21 provides our assessment of VA’s  compliance with OMB’s requirements.
EPA does not plan to further consolidate its four primary data centers.  Instead, the agency plans to focus its consolidation efforts on achieving  efficiencies via virtualization within those four centers. However, the  agency’s asset inventory and consolidation plan are not complete. In its  inventory, the agency provides complete information for 3 of the key  elements and provides partial information for the remaining element.  Additionally, in its consolidation plan, EPA provides complete information  for 5 of the 10 elements evaluated, provides partial information for 1  element, and does not provide information for the remaining 4 elements.  Table 22 provides our assessment of EPA’s compliance with OMB’s  requirements.
GSA plans to consolidate from 15 data centers to 3 by fiscal year 2015.  However, the agency’s asset inventory and consolidation plan are not  complete. In its inventory, the agency provides complete information for 2  of the key elements and provides partial information for the remaining 2  elements. Additionally, in its consolidation plan, GSA provides complete  information for 8 of the 10 elements evaluated, provides partial  information for 1 element, and does not provide information for the  remaining element. Table 23 provides our assessment of GSA’s  compliance with OMB’s requirements.
NASA plans to consolidate from 79 data centers to 57 by fiscal year 2015.  However, the agency’s asset inventory and consolidation plan are not  complete. In its inventory, the agency provides partial information for 3 of  the key elements and does not provide information for the remaining  element. Additionally, in its consolidation plan, NASA provides complete  information for 4 of the 10 elements evaluated, provides partial  information for 1 element, and does not provide information for the  remaining 5 elements. Table 24 provides our assessment of NASA’s  compliance with OMB’s requirements.
NSF owns and operates one data center and utilizes one commercial  data center. The agency aims to transition all operations to one  commercial data center by fiscal year 2014. The agency’s asset inventory  is complete, but its consolidation plan is not. Specifically, NSF provides  complete information for 7 of the 10 elements evaluated, and does not  provide information for the remaining 3 elements. Agency officials stated  that they have a master schedule and risk management plan, but  acknowledged that they did not provide this information to OMB as part of  their consolidation plan. Table 25 provides our assessment of NSF’s  compliance with OMB’s requirements.
NRC plans to consolidate from three existing data centers into one new  center by fiscal year 2013. However, the agency’s asset inventory and  consolidation plan are not complete. In its inventory, the agency provides  complete information for 3 of the key elements and provides partial  information for the remaining element. Additionally, in its consolidation  plan, NRC provides complete information for 4 of the 10 elements  evaluated, provides partial information for 1 element, and does not  provide information for the remaining 5 elements. Table 26 provides our  assessment of NRC’s compliance with OMB’s requirements.
OPM does not plan to further consolidate its one data center. Instead, the  agency plans to continue to examine and execute ways to improve the  efficiency of its IT operations, such as through virtualization. However, the  agency’s asset inventory and consolidation plan are not complete. In its  inventory, the agency provides complete information for 1 key element  and partial information for the remaining 3 elements. Additionally, in its  consolidation plan, OPM provides complete information for 5 of the 10  elements evaluated, provides partial information for 1 element, and does  not provide information for the remaining 4 elements. Table 27 provides  our assessment of OPM’s compliance with OMB’s requirements.
SBA plans to reduce its number of data centers from four to two by fiscal  year 2015. However, the agency’s asset inventory and consolidation plan  are not complete. In its inventory, the agency provides complete  information for 1 key element and partial information for the remaining 3  elements. Additionally, in its consolidation plan, SBA provides complete  information for 5 of the 10 elements evaluated, partial information for 1  element, and does not provide information for the remaining 4 elements.  Table 28 provides our assessment of SBA’s compliance with OMB’s  requirements.
SSA does not plan to further consolidate its two data centers. In line with  the goals of the FDCCI, the agency plans to improve the efficiency,  performance, and stability of its IT infrastructure by reducing the number  of its remote operations control centers. However, the agency’s asset  inventory and consolidation plan are not complete. In its inventory, the  agency provides partial information for all 4 key elements. Additionally, in  its consolidation plan, SSA provides complete information for 4 of the 10  elements evaluated, partial information for 1 element, and does not  provide information for the remaining 5 elements. Table 29 provides our  assessment of SSA’s compliance with OMB’s requirements.
USAID plans to consolidate from two data centers into one by fiscal year  2015. However, the agency’s asset inventory and consolidation plan are  not complete. In its inventory, the agency provides complete information  for 1 key element and partial information for the remaining 3 elements.  Additionally, in its consolidation plan, USAID provides complete  information for 5 of the 10 elements evaluated, provides partial  information for 1 element, and does not provide information for the  remaining 4 elements. Table 30 provides our assessment of USAID’s  compliance with OMB’s requirements.
section_title: Appendix IV: OMB-Defined Approaches for Consolidating Data Centers
paragraphs: In its guidance on data center consolidations, OMB identified four  approaches for agencies to consider while evaluating the feasibility of  consolidating the individual systems found within each data center. OMB  directed agencies to specify which of these approaches was to be utilized  for each data center. The four approaches are as follows:    Decommissioning: the system is no longer in use or it is redundant  and will be decommissioned.   Consolidation: the system will be consolidated onto a shared  infrastructure with other similar systems.   Cloud computing: the system will be migrated to or replaced by  Internet-based services and resources.   Virtualization: the system will be migrated to a virtual machine  environment.
In response to OMB’s guidance, agencies reported that they will pursue a  variety of consolidation approaches. Agency-specific examples of how  these approaches will be employed are provided below.
subsections: section_title: Decommissioning
paragraphs: Agencies may choose to decommission their underutilized physical  servers as a part of their data center consolidation plans. For example,  EPA plans to decommission more than 900 physical servers by 2015.  Also, Labor plans to decommission unused servers and storage hardware  and replace inefficient hardware with “green IT” hardware. Further, GSA  plans to identify and decommission inefficient and underutilized legacy  servers and equipment.
Agencies can also choose to decommission an entire data center by  moving to an outsourced data center or reducing the number of physical  assets. For example, as part of its data center consolidation initiative,  NSF plans to decommission its single data center by fiscal year 2014 and  to move to a commercial facility. Transportation plans to decommission  data centers that spread out across multiple buildings and reduce the  department’s number of data centers by approximately 25 percent by the  close of fiscal year 2015.
section_title: Consolidation
paragraphs: Consolidation is a means of combining workload onto fewer computers or  concentrating data processing into fewer physical facilities. Physically  moving data processing equipment from multiple locations to a lesser  number of locations can assist agencies in reaching consolidation goals,  such as reducing the cost of data center hardware, software, and  operations, in addition to real estate and energy costs. For example, DHS  has 43 principal data centers, all of which will be moved into one of two  enterprise data centers by the end of fiscal year 2014. In addition, NASA  plans to consolidate from 79 data centers to 57 by fiscal year 2015.
section_title: Cloud computing
paragraphs: Cloud computing is an emerging form of computing that relies on Internet- based services and resources to provide computing services to  customers, while freeing them from the burden and costs of maintaining  the underlying infrastructure. This approach is a form of delivering IT  services that takes advantage of several broad evolutionary trends,  including the use of virtualization; the decreased cost and increased  speed of networked communications, such as the Internet; and overall  increases in computing power. Examples of cloud computing include  Web-based e-mail applications and common business applications that  are accessed online through a browser instead of through a local  computer. Several agencies are considering both cloud computing and  virtualization as a means of achieving their consolidation goals. For  example, SBA has plans to migrate commodity computing services such  as Web hosting and messaging to cloud solutions. We have recently  reported on challenges associated with the implementation of cloud  computing.
section_title: Virtualization
paragraphs: Virtual machines can be stored as files, making it possible to save a  virtual machine and move it from one physical server to another.  Virtualization is often used as part of cloud computing. For example, one  Defense component reports that 45 percent of all server operating  environments supporting customer workload in its data centers have been  virtualized. State plans to reduce its environmental impact by hosting 70  percent of the department’s servers on virtual infrastructure by 2015.  Also, NRC has virtualized 41 Windows-based applications and has  identified 50 additional applications to be virtualized by 2013.
section_title: Appendix V: Comments from the Department of Commerce
section_title: Appendix VI: Comments from the Department of Defense
section_title: Appendix VII: Comments from the Department of Education
section_title: Appendix VIII: Comments from the Department of Energy
section_title: Appendix IX: Comments from the Department of Health and Human Services
section_title: Appendix X: Comments from the Department of Homeland Security
section_title: Appendix XI: Comments from the Department of the Interior
section_title: Appendix XII: Comments from the Department of Labor
section_title: Appendix XIII: Comments from the Department of State
section_title: Appendix XIV: Comments from the Department of the Treasury
section_title: Appendix XV: Comments from the Department of Veterans Affairs
section_title: Appendix XVI: Comments from the Environmental Protection Agency
section_title: Appendix XVII: Comments from the General Services Administration
section_title: Appendix XVIII: Comments from the National Aeronautics and Space Administration
section_title: Appendix XIX: Comments from the Nuclear Regulatory Commission
section_title: Appendix XX: Comments from the Social Security Administration
section_title: Appendix XXI: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Colleen Phillips (Assistant  Director), Neil Doherty, Rebecca Eyler, Nancy Glover, Dave Hinchman,  Linda Kochersberger, and Jessica Waselkow made key contributions to  this report.