Data Center Consolidation: Agencies Need to Complete Inventories and Plans to Achieve Expected Savings
Released: 2011-07-26T00:00:00

Letter

Background

While the term “data center” can be used to describe any room used for the purpose of processing or storing data, OMB defines a data center as a room that is greater than 500 square feet, that is used for processing or storing data, and that meets stringent availability requirements. Other facilities are classified as “server rooms,” which are typically less than 500 square feet and “server closets,” which are typically less than 200 square feet.

According to OMB, the number of federal data centers grew from 432 in 1998 to 2,094 in July 2010. Operating such a large number of centers places costly demands on the government. While the total annual federal spending associated with data centers has not yet been determined, OMB has found that operating data centers is a significant cost to the federal government, including hardware, software, real estate, and cooling costs. For example, according to the Environmental Protection Agency (EPA), the electricity cost to operate federal servers and data centers across the government is about $450 million annually. According to the Department of Energy (Energy), data center spaces can consume 100 to 200 times as much electricity as standard office spaces. Reported server utilization rates as low as 5 percent and limited reuse of these data centers within or across agencies lends further credence to the need to restructure federal data center operations to improve efficiency and reduce costs. In 2010, the Federal Chief Information Officer (CIO) reported that operating and maintaining such redundant infrastructure investments was costly, inefficient, and unsustainable.

OMB and the Federal CIO Established the Federal Data Center Consolidation Initiative

Concerned about the size of the federal data center inventory and the potential to improve the efficiency, performance, and environmental footprint of federal data center activities, in February 2010 OMB, under the direction of the Federal CIO, announced the Federal Data Center Consolidation Initiative (FDCCI). This initiative’s four high-level goals are to  promote the use of “green IT” by reducing the overall energy and real estate footprint of government data centers; reduce the cost of data center hardware, software, and operations; increase the overall IT security posture of the government; and shift IT investments to more efficient computing platforms and technologies.

As part of FDCCI, OMB required 24 departments and agencies that participate on the Chief Information Officers Council (see table 1) to submit a series of documents that ultimately resulted in a data center consolidation plan.

Specifically, the departments and agencies were to provide the following:  An initial asset inventory (due April 30, 2010), which was to provide a high-level understanding of the scale and size of existing data centers, IT infrastructure assets, and applications supported by the data centers.  An initial data center consolidation plan (due June 30, 2010), which was to identify potential areas for consolidation, areas where optimization through server virtualization or cloud computing alternatives could be used, and a high-level roadmap for transitioning to the consolidated end-state architecture.  A final asset inventory baseline (due July 30, 2010), which was to contain more detailed information and serve as the foundation for developing the final data center consolidation plans. The final inventory was also to identify the consolidation approach to be taken for each data center.  A final data center consolidation plan (due August 30, 2010), which was to be incorporated into the agency’s fiscal year 2012 budget and was to include a technical roadmap and approach for achieving the targets for infrastructure utilization, energy efficiency, and cost efficiency.

In October 2010, OMB reported that all of the agencies had submitted their plans and that there were 2,094 federal data centers as of July 2010. OMB announced plans to monitor agencies’ consolidation activities on an ongoing basis as part of the annual budget process. Further, starting in fiscal year 2011, agencies will be required to provide an annual updated data center asset inventory at the end of every third quarter and a consolidation progress report at the end of every fourth quarter.

To manage the initiative, OMB designated two agency CIOs as executive sponsors to lead the effort within the Chief Information Officers Council. Additionally, the General Services Administration (GSA) has established the FDCCI Program Management Office, whose role is to support OMB in the planning, execution, management, and communication for FDCCI. In this role, GSA collected the responses to the four document deliveries and reviewed the submissions for completeness and reasonableness. GSA also sponsored three workshops on the initiative for agencies and facilitated a peer review of the initial and final data center consolidation plans.

OMB’s IT Reform Plan Sets Important Milestones for Data Center Consolidation

In December 2010, OMB published its 25-Point Implementation Plan to Reform Federal Information Technology Management as a means of implementing IT reform in the areas of operational efficiency and large- scale IT program management. Among the 25 initiatives, OMB has included two goals that relate to data center consolidation: 1. By June 2011, complete detailed implementation plans to consolidate at least 800 data centers by 2015. 2. By June 2012, create a governmentwide marketplace for data center availability.

To accomplish its first goal, OMB required each FDCCI agency to identify a senior, dedicated data center consolidation program manager. It also launched a Data Center Consolidation Task Force comprised of the data center consolidation program managers from each agency. OMB officials stated that this task force is critical to driving forward on individual agency consolidation goals and to meeting the overall federal target of closing a minimum of 800 data centers by 2015. To that end, in April 2011, OMB announced plans to close 137 data centers by the end of December 2011. OMB also plans to launch a publicly-available dashboard for observing agencies’ consolidation progress, but this has not yet been completed.

To accomplish its second goal, OMB and GSA plan to create a governmentwide marketplace by June 2012 that will better utilize spare capacity within operational data centers. This online marketplace is intended to match agencies that have extra capacity with agencies with increasing demand, thereby improving the utilization of existing facilities. The marketplace will help agencies with available capacity promote their available data center space. Once agencies have a clear sense of the existing capacity landscape, they can make more informed consolidation decisions.

GAO Has Previously Reported on Federal Data Center Consolidation Efforts

We have previously reported on OMB’s efforts to consolidate federal data centers. In March 2011, we reported on the status of the FDCCI and noted that data center consolidation makes sense economically and as a way to achieve more efficient IT operations, but that challenges exist. For example, agencies face challenges in ensuring the accuracy of their inventories and plans, providing upfront funding for the consolidation effort before any cost savings accrue, integrating consolidation plans into agency budget submissions (as required by OMB), establishing and implementing shared standards (for storage, systems, security, etc.), overcoming cultural resistance to such major organizational changes, and maintaining current operations during the transition to consolidated operations. We further reported that mitigating these and other challenges will require commitment from the agencies and continued oversight by OMB and the Federal CIO.

Agencies Established Consolidation Plans, but Incomplete Inventories and Missing Elements Undermine Expected Savings

To help agencies plan their consolidations, OMB issued guidance on the required content of data center inventories and consolidation plans. Specifically, the inventories were to include descriptions of the assets present within individual data centers, as well as information about the physical data center itself. The consolidation plans were to address key elements, including goals, approaches, schedules, cost-benefit calculations, and risk management plans.

As required, 23 of the 24 agencies submitted their inventories and consolidation plans by the end of September 2010; the remaining agency explained that consolidation was not applicable to them. However, of the 23 reporting agencies, all but one of the inventories and all of the plans are missing key elements. For example, 14 agencies do not provide a complete listing of data centers and 15 do not provide a complete listing of software assets in their inventories. Further, OMB did not require that agencies verify these inventory data. Additionally, in their consolidation plans, 20 agencies do not provide a master schedule, 12 agencies do not address cost-benefit calculations, and 9 do not address risk management. Several agency officials noted that they had difficulty completing their inventories and plans within OMB’s timelines. Other agencies reported trouble with identifying either required information for the plans or data on the assets within their data centers. Until these inventories and plans are complete, agencies may not be able to implement their consolidation activities or to realize expected cost savings. Moreover, without an understanding of the validity of agencies’ consolidation data, OMB cannot be assured that agencies are providing a sound baseline for estimating consolidation savings and measuring progress against those goals.

Agencies Plan Significant Facility Reductions and Cost Savings

In their consolidation plans, agencies identified goals for reducing the number of data center facilities and these facilities’ related costs. Specifically, the 23 reporting agencies identified 1,590 data centers as of April 2011, and established goals for reducing that number by 652 centers by the end of fiscal year 2015.

Most federal departments and agencies also estimated cost savings over time. Specifically:  Fourteen agencies reported savings totaling about $700 million between fiscal years 2011 and 2015; however, actual savings may be even higher because 12 of these agencies’ estimates were incomplete. For example, 11 agencies included expected energy savings and reductions in building operational costs, but not savings from other sources, such as equipment reductions.  Two agencies expect to accrue net savings after fiscal year 2015.  Two agencies do not expect to attain net savings from their consolidation efforts.  Five agencies did not provide estimated cost savings; however, two of these agencies suggested that they plan to develop cost-benefit analyses in the future.

Asset Inventories Are Not Complete or Verified

As part of the data center consolidation initiative, OMB required agencies to provide an inventory of data center assets. This inventory is to address four key elements: (1) IT software assets; (2) IT hardware assets and their utilization; (3) IT facilities, energy, and storage; and (4) geographic location and real estate. According to OMB’s guidance, the information is to be organized by data center. For example, in the IT software area, agencies are to report by data center on each major and nonmajor system present in the center. For each identified system, the agency is to report the associated support platforms, servers and computers, and proposed consolidation approach (i.e., decommissioning, consolidation, cloud computing, or virtualization). Table 2 provides a detailed description of each of the four key elements.

When collecting data, it is important to have assurance that the data are accurate. We have previously reported on the need for agencies, when providing information to OMB, to explain the procedures used to verify their data. Specifically, agencies should ensure that reported data are sufficiently complete, accurate, and consistent, and also identify any significant data limitations. Explaining the limitations of information can provide a context for understanding and assessing the challenges agencies face in gathering, processing, and analyzing needed data. Such a presentation of data limitations can also help identify the actions needed to improve the agency’s ability to measure its performance. More recently, we have reiterated the importance of providing OMB with complete and accurate data and the possible negative impact of that data being missing or incomplete.

Only 1 of the 23 agency data center inventories contains complete data in all four of the required elements. Specifically, while many agencies provide partial inventory data,  one agency provides complete information in all four areas, five agencies provide complete information in three of the four areas,  one agency provides complete information for two of the areas,  eight agencies have complete information for only one area, and  eight agencies do not have any complete areas in their inventories.

Figure 1 provides an assessment of the completeness of agencies’ inventories, by key element, and a discussion of the analysis of each area follows the figure.

IT software assets. Eight agencies provide complete information on their software assets; 14 agencies provide partial information; and 1 agency did not provide information. For example, the Small Business Administration (SBA) provides information on its data center systems, their technical dependencies on platforms and servers, and consolidation approaches for its systems; while GSA provides information on its data center systems and a consolidation approach for each system, but provides only partial information on each system’s technical dependencies on platforms and servers. Additionally, Energy provides only partial information on the systems in its data center, the systems’ technical dependencies, and consolidation approaches for those systems.

IT hardware assets and utilization. Nine agencies provide complete information on their IT hardware assets and the utilization of those assets and 14 provide partial information. For example, EPA provides complete information on maximum and average server utilization, as well as counts of its physical servers, virtual hosts, and virtual operating systems; while SBA provides complete information on counts of its physical servers, virtual hosts, and virtual operating systems, but only partial information on maximum and average server utilization. Another 7 agencies, including the Departments of Defense (Defense), Homeland Security (DHS), and Transportation (Transportation), provide partial information on their maximum and average server utilization, and on their counts of physical servers, virtual hosts, and virtual operating systems.

IT facilities, energy, and storage. Three agencies provide complete information on their IT facilities, energy, and storage, while 20 provide partial information. For example, the Department of State (State) includes all the required information, while the Department of Education (Education) provides complete information on its annual data center operational cost, total rack count, and storage information, but only provides partial information on its annual data center electricity cost and total electricity usage. Also, the National Aeronautics and Space Administration (NASA), has partial information on its total rack count and does not provide any information for the other required parts of this element.  Geographic location and real estate. Nine agencies provide complete information on their data center locations, while 14 provide partial information. For example, the Office of Personnel Management (OPM) provides complete information on its number of data centers and the gross floor area of those centers, but does not provide information on the number of server rooms and closets and the gross floor area of those facilities. Other agencies such as Energy and the Department of Labor (Labor) only provide partial information on their number of centers, server rooms, and closets, and the gross floor area of those facilities.

Because agency goals are intended to be built on the information provided by the inventories, agencies cannot ensure the reliability of their savings and utilization forecasts until the underlying inventories have been completed.

While it is important that reported data are sufficiently complete, accurate, and consistent, OMB’s guidance on agency inventories does not require agencies to document what they did to verify their data, or to disclose any limitations on that data. Nonetheless, several agencies took informal steps to validate their data. For example, Department of Agriculture (Agriculture) officials stated that they interviewed staff who submitted inventory information and conducted on-site visits of data centers. Additionally, Department of Commerce (Commerce) officials reported that they reviewed the inventory data and clarified missing or suspect entries with those who submitted the information. Also, a Department of the Treasury (Treasury) official stated that there were two rounds of data verification and that Treasury bureaus were sometimes asked to verify submitted information. However, officials from other agencies, such as Defense, Energy, and NASA, confirmed that their inventories had not been verified. Further, in some cases, such as with the Department of Health and Human Services (HHS) and NASA, agency officials reported that their inventory information was estimated.

Notwithstanding agencies’ informal verification efforts, complete, accurate, and consistent performance information will be important to OMB to guide its decisions on how best to oversee federal data center consolidations. Without an understanding of the validity and limitations on agencies’ data, OMB cannot be assured that agencies are providing a sound baseline for estimating savings and measuring progress against their goals.

Agencies Established Consolidation Plans, but Work Remains on Key Elements

In addition to the agencies’ inventories, OMB required agencies to establish consolidation plans that address key elements, including quantitative goals, qualitative impacts, approach, scope, timeline, and master schedule, as well as summaries of a cost-benefit analysis, performance metrics, risk management, and communications planning. OMB noted the importance of agencies’ consolidation plans in providing a technical road map and approach for achieving specified targets for infrastructure utilization, energy efficiency, and cost efficiency. Table 3 provides a detailed description of each of these elements.

While 23 agencies submitted consolidation plans to OMB, selected elements are missing from each plan. For example, 22 agencies provide complete information on their qualitative impacts, but only 6 provide complete information on their quantitative goals. Further, while all 23 agencies specify their consolidation approach, only 5 indicate that a cost- benefit analysis was performed for the consolidation initiative. In many cases, agencies submitted some, but not all, of the required information. Figure 2 provides an assessment by element, and a discussion of each element follows the figure. A detailed summary of the agencies’ status of completion of each key element is provided in appendix II. In addition, this information is provided for each agency in appendix III.  Quantitative goals. Six agencies provide complete savings and utilization forecasts and 17 agencies provide partial forecasts. For example, Defense’s savings and utilization forecasts are incomplete, while Treasury and the Social Security Administration (SSA) provide complete savings forecasts, but incomplete utilization forecasts. Some agencies identified reasons for not having completed these forecasts. For example, Treasury’s plan states that the department’s savings and utilization targets do not include demand from new organizations required under recent legislation and that the plan will be updated as further information becomes available. The plan also notes that the forecasts could change when the department completes associated cost-benefit analyses. Additionally, NASA’s plan states that the agency is performing an assessment of its assets to form an accurate baseline, on which actual targets for reduction can be predicted. This plan also notes that as the agency’s asset and inventory information is improved, NASA will evaluate opportunities to further consolidate applications and virtualize operating systems.  Qualitative impacts. Twenty-two agencies describe the qualitative impacts of their consolidation initiatives and 1 agency does not. For example, Agriculture’s plan describes goals such as reducing overall energy use and reducing the real estate footprint for data centers. Additionally, HHS reports that the consolidation effort will result in more efficient monitoring of data center power. Finally, the U.S. Agency for International Development (USAID) describes goals such as optimizing use of IT funding by increasing efficiencies and increasing the availability of resources and systems to the user community. However, Education does not provide any qualitative impacts. Although the department links its consolidation plan to a federal strategic sustainability plan, the sustainability plan does not contain qualitative impacts such as those required by OMB.  Summary of consolidation approach. All 23 agencies include a summary of the agencies’ proposed consolidation approaches. For example, Commerce describes five approaches that will support the department’s FDCCI goals: consolidating and decommissioning data centers, increasing server virtualization and IT equipment utilization, moving to cloud computing, acquiring green products and services, and promoting “green IT.” Similarly, the Department of the Interior (Interior) provides four approaches to help realize its consolidation goals: decommissioning, consolidation, cloud computing, and server and storage virtualization. For a more detailed discussion of alternative data center consolidation approaches, see appendix IV.  Scope of consolidation. Nineteen agencies’ plans include a well- defined scope for data center consolidation, 2 provide partial information on the scope of their consolidation efforts, and 2 do not provide this information. Specifically, the agencies that provide this information list the data centers included in the consolidation effort and what consolidation approach will be taken for the systems within each center (i.e., decommissioning, consolidation, cloud computing, or virtualization). For example, the Department of Veterans Affairs (VA) lists 87 data centers and Labor lists 20 data centers, in which all of the systems will be consolidated, decommissioned, or virtualized. Alternatively, Energy has not yet determined what action will be taken for each of its facilities and Interior identifies the total numbers of centers to be retained or expanded, but does not describe the consolidation approach for individual data centers. Two agencies, the Department of Justice (Justice) and NASA, are still working to determine which of their data centers are to be consolidated.  High-level timeline. Twenty agencies include a high-level timeline for consolidation efforts, 1 agency includes partial information on its timelines, and 2 do not provide timelines. For example, Labor and VA both provide the fiscal years in which every data center listed will be consolidated and the National Science Foundation (NSF) states what year the agency’s primary data center will be decommissioned and replaced with private and public cloud services. In contrast, Defense only describes broad goals to be accomplished by fiscal year 2013 and does not include specific milestones for each data center. Further, NASA does not include this information in its plan and notes that the agency is still working to determine its data center inventory.  Performance metrics. Six agencies identify specific performance metrics for their consolidation programs, 4 agencies provide partial information on their metrics, and 13 agencies did not identify specific metrics. For example, both Transportation and GSA specify metrics such as savings in energy consumption, cost variance, and schedule variance. Alternatively, State reports that the department’s data center consolidation program maintains metrics at both the system and process performance levels, but does not provide any specifics as to the nature of those metrics. Further, although Defense does not provide metrics at the department level, the Air Force has developed a method to provide such measures.  Master program schedule. Three agencies reference a completed master program schedule, and 20 do not. For example, while Agriculture, DHS, and Interior discuss their master schedules, other agencies, such as Commerce and HHS do not. Some agencies, such as Defense, Labor, and the Nuclear Regulatory Commission (NRC), plan to develop them in the future.  Cost-benefit analysis. Five agencies provide a cost-benefit analysis that encompasses their entire consolidation initiative, 6 agencies provide only selected elements of a cost-benefit analysis, and 12 agencies do not provide a cost-benefit analysis. For example, DHS details full annualized investment and savings estimates through fiscal year 2015, while other agencies, such as State and OPM, provide only partial information. Specifically, State acknowledges that not all costs are accounted for in its analysis and OPM reports that its analysis is preliminary. Additionally, Commerce provides costs and savings for several data center consolidations but acknowledges that estimates cannot be provided for all of the department’s planned consolidation initiatives. Eight of the agencies that do not provide a cost-benefit analysis, such as HHS, Justice, and USAID, plan to conduct one in the future.  Risk management. Fewer than half of the agencies both reference a consolidation risk management plan and require that risks be tracked. For example, HHS discusses its approach to risk management and identifies a series of technical, security, funding, and management risks and provides a mitigation strategy for each. Additionally, VA describes a five-phase approach to risk management that includes identifying and monitoring risks. However, Education requires that risks be tracked, but does not reference the existence of an actual risk management plan. Nine agencies do not reference a risk management plan or requirements for tracking risks.  Communications plan. Eighteen agencies consider a communications strategy for the agencies’ consolidation initiatives, and 5 agencies do not. For example, Energy describes a series of coordinated activities that are intended to support the consolidation effort. Additionally, NASA details its approach to consolidation coordination and communication and SBA details individual communication responsibilities among consolidation stakeholders. However, Treasury and NRC do not describe such a communications strategy.

When asked about the elements missing from their plans, many agency officials stated that they completed what they could within the timelines provided by OMB. Several agency officials noted that it was difficult to obtain all of the required data from component agencies, while others reported that their data collection efforts were made more difficult by OMB’s tight time frames and changes in templates and guidance. Moreover, officials from two agencies stated that some of the information contained in their plans had been estimated. However, OMB has not required agencies to complete the missing elements or to resubmit their final plans. According to an OMB official, agencies have been instructed to move forward with their consolidation initiatives, and as noted earlier, OMB intends to monitor the agencies’ progress annually.

We have previously reported that without a clear description of the strategies and resources an agency plans to use in meeting its goals, it will be difficult to assess the likelihood of the agency’s success in achieving its intended results. In the absence of completed consolidation plans, agencies run the risk of moving forward on their respective initiatives with, among other things, poorly defined approaches and outcomes. Without this information, agencies may not realize anticipated cost savings, improved infrastructure utilization, and energy efficiency.

Agencies Face Multiple Challenges in Consolidation

In preparing agencies for the data center consolidation initiative, OMB held workshops that, among other things, discussed challenges that agencies might face so that they could anticipate and mitigate them. In addition, agencies identified multiple challenges they are facing during data center consolidations. These include challenges related to the data center consolidation initiative as well as those that are cultural, funding related, operational, and technical. Some challenges are more common than others. Specifically, the number of agencies reporting a particular challenge range from 1 agency to 19 agencies. Table 4 details the reported challenges as well as the numbers of agencies experiencing that challenge. The table is followed by a discussion of the most prevalent challenges.

Agencies reported nine challenges that are specific to OMB’s data center consolidation initiative, including meeting tight FDCCI deadlines and obtaining power usage data as required by OMB. Specifically, 19 agencies reported that obtaining power usage data was a challenge. For example, a Commerce official stated that while more than half of the agency’s data centers have power consumption figures or costs associated with power usage, some of the agency’s facilities do not have metering capabilities for power consumption and that the agency has some server rooms that lack metering at the equipment. Similarly, Labor’s Deputy CIO stated that the department does not have metering for data centers. Consequently, the agency used best practices to estimate how much power is being used at data centers. In addition, 11 agencies found that the tight FDCCI deadlines were a challenge. For example, Labor’s Deputy CIO stated that the time frames were overly aggressive and did not allow the agency to provide the information OMB requested or to complete the planning that is necessary for such an important undertaking. An Energy official stated that there is no quick path to consolidation and that the agency was faced with the decision of either moving forward with inadequate inventory information or taking more time to make decisions. The official stressed that the agency would like to consolidate in the correct manner.

Agencies reported four cultural challenges to data center consolidation, including accepting cultural change and implementing consolidation in an organizational structure not geared towards consolidation (i.e., a decentralized enterprise). The most prevalent challenge was acceptance of cultural change, with 15 agencies reporting it as a challenge. For example, an Agriculture official stated that there is a challenge in addressing cultural change surrounding data center consolidation. With data center consolidation, systems personnel may not be in the same location as the data. To address this, Agriculture refined its communications plan so that lessons learned can be passed on to other staff. An EPA official noted the agency experienced this challenge when employees were reluctant to cede control of resources under their immediate control. EPA mitigated that challenge by building relationships with stakeholders early on. In addition, 8 agencies reported that implementing the consolidation was challenged by their organizational structures. For example, a Justice official stated that the agency uses a federated IT approach within which the departmentwide CIO’s office has primary responsibility for architecture, common infrastructure, and standards decisions, while each component IT department has primary responsibility for application resource decisions. According to this official, such a federated approach offers Justice’s components more autonomy when making decisions, but creates obstacles for departmentwide efforts such as FDCCI. Further, an Energy official cited the department’s decentralized environment as a challenge in being able to collect data center inventory information.

The data center consolidation initiative is supposed to result in cost savings, but multiple agencies reported challenges in funding their initiatives. For example, a Transportation official reported that the need for upfront funding for consolidation efforts was a challenge. A NASA official stated that the agency spent approximately $1.5 million on an asset management tool to assist the agency in creating its inventory. Further, a State official noted the challenge of having to fund the consolidation efforts long before cost savings will be realized. In addition, 9 agencies reported that identifying cost savings for consolidation efforts was a challenge. For example, an Energy official stated that it was too early in the consolidation process for Energy to be able to quantify cost savings, since the agency does not have data on cost or exactly which data centers will be closed. A State official stated that it is difficult to estimate cost savings since the department does not have information on power usage for all facilities. Further, a Treasury official noted challenges in identifying cost savings, particularly because a reduction in utilized square footage at a facility does not mean that the leasing agency will issue a refund check on the lease. Similarly, building managers at private facilities will typically not issue a refund if an agency begins using less energy.

Agencies reported eight operational challenges to data center consolidation, including maintaining services during the consolidation transition and implementing cloud computing. Nine agencies reported that maintaining services during the consolidation transition is a challenge. For example, a Labor official stated that keeping the business running during the transition is a big concern of the agency. Three agencies reported that moving to cloud computing was a challenge. For example, a Commerce official stated that the agency’s biggest challenge was how to implement cloud computing. This official cited the need to investigate how private and public cloud computing will fit into the agency’s mission, and to determine how to manage security issues surrounding cloud computing.

Agencies reported 10 technical challenges to data center consolidation, including maintaining the appropriate level of system security and planning the migration strategy. While agencies reported more technical challenges than any other type of challenge, these challenges are more diverse, with fewer agencies experiencing each individual challenge. Three agencies reported that maintaining the appropriate level of system security was a challenge. For example, an OPM official stated that one of the challenges faced was the need to maintain personally identifiable information while exploring options such as cloud computing. In addition, a State official identified the challenge of including classified servers in the consolidation initiative. Two agencies reported that planning the migration strategy was a challenge. For example, an SSA official pointed out the difficulty in scheduling migration across approximately 1,500 field offices.

One approach agencies can use to manage challenges such as the ones listed above is through formal risk management processes. However, as noted in the prior section, less than half of the agencies included a discussion of risk management in their data center consolidation plans.

Lessons Learned by States Can Be Leveraged to Mitigate Challenges at the Federal Level

We have previously reported on the importance of using lessons learned—a principal component of an organizational culture committed to continuous improvement. Sharing such information serves to communicate acquired knowledge more effectively and to ensure that beneficial information is factored into planning, work processes, and activities. Lessons learned also provide a powerful method of sharing good ideas for improving work processes, facility or equipment design and operation, quality, safety, and cost-effectiveness.

OMB posted lessons learned by three states on its data center consolidation Web page for federal agencies to review. However, there is more that agencies can learn. Many state governments have undertaken data center consolidation initiatives in recent years. Although they have encountered unique challenges, they have also encountered challenges similar to those reported by federal agencies. Specifically, the National Association of State CIOs and a literature search identified 20 states that reported on challenges they faced, or lessons they learned, from their data center consolidation initiatives. Of these, 19 reported lessons learned that could be leveraged at the federal level. For example, officials from North Carolina reported that organizations are typically concerned that by consolidating data centers, they will lose control of their data, service levels will decline, or costs will rise. The state learned that to help mitigate this during the process of consolidation, the organizations’ concerns should be documented, validated, and addressed. In another example, a West Virginia official reported that since the state had no funding for a consolidation, it had to be creative in executing the consolidation. The state used the natural aging cycle of hardware to force consolidation; that is, when a piece of hardware was ready to be replaced, the applications and software were put onto a consolidated server. As a final example, two states reported lessons learned that could be applied to the challenge of providing a quality asset inventory. Officials from Utah and Texas emphasized the importance of having an accurate inventory of all equipment that could be impacted by the project. The official from Texas added that it is important to have a third party collect technical data across agencies, and that it is beneficial to have this work completed by outside parties, in order to ensure objectivity and consistency. Table 5 identifies lessons learned by states that could be applied by federal agencies.

Conclusions

With agencies reporting having almost 1,600 federal data centers, OMB’s goal of consolidating 800 centers by 2015 is ambitious. To its credit, OMB has established an accountability infrastructure through its data center consolidation task force composed of representatives from each of the participating agencies. OMB and federal agencies have also taken important steps to reduce the number and increase the efficiency of the federal data centers. However, only one agency has completed its required data center asset inventory, no agencies have completed their consolidation plans, and OMB has not required that agency inventory information be verified. Despite these limitations, OMB has instructed agencies to move forward with their plans. Moving forward to consolidate obviously redundant or underutilized centers is warranted—and should result in immediate cost savings and increased efficiency. However, without a complete asset inventory and a comprehensive plan, agencies are at increased risk that they will be ill-prepared to manage such a significant transformation. This could slow the consolidations and reduce expected savings and efficiencies.

In moving ahead in their consolidation efforts, agencies are encountering challenges, including those that are technical, operational, and cultural in nature. Some state governments have also engaged in data center consolidation initiatives and dealt with similar obstacles in doing so. By virtue of these experiences, these states can offer insights and suggestions that federal agencies can use to mitigate their challenges and risks. In doing so, agencies will be better positioned to address their consolidation goals and to meet OMB’s goals for reducing the number and cost of federal data centers.

Recommendations for Executive Action

To better ensure that the federal data center consolidation initiative improves governmental efficiency and achieves cost savings, we are making four recommendations to OMB. Specifically, we are recommending that the Director of the Office of Management and Budget direct the Federal Chief Information Officer to require that agencies, when updating their data center inventories in the third quarter of each fiscal year, state what actions have been taken to verify the inventories and to identify any limitations of this information; require that agencies complete the missing elements in their respective plans and submit complete data center consolidation plans, or provide a schedule for when they will do so, by September 30, 2011; require agencies to consider consolidation challenges and lessons learned when updating their plans; and  utilize the existing accountability infrastructure by requiring the Data Center Consolidation Task Force to assess agency consolidation plans to ensure they are complete and to monitor the agencies’ implementation of their plans.

In addition, we are making two recommendations to each of the department secretaries and agency heads of the 23 departments and agencies participating in the federal data center consolidation initiative. Specifically, we are recommending that the secretaries and agency heads  direct their component agencies and their data center consolidation program managers to complete the missing elements in their respective data center consolidation inventories and plans; and require their data center consolidation program managers to consider consolidation challenges and lessons learned when updating their consolidation plans.

Agency Comments and Our Evaluation

We received comments on a draft of our report from OMB and the 23 agencies to which we made recommendations. Most agencies generally agreed with our recommendations. Specifically, in commenting on the draft, 15 agencies agreed with our recommendations; 4 agreed with the report’s content or findings, but offered no comments on the recommendations; 3 offered no comments on the report’s findings or recommendations; and Defense and SSA both did not agree with one of our recommendations, but agreed with the second. Agencies also provided technical comments, which we incorporated as appropriate. Each agency’s comments are discussed in more detail below.

In comments provided via e-mail, an OMB official from the General Counsel Office wrote that the agency generally agreed with our report. The agency offered no comments on our recommendations.

In comments provided via e-mail, Agriculture’s CIO agreed with our recommendations and noted that our assessment of USDA’s inventory was accurate.

In written comments, the Secretary of Commerce concurred with the general findings as they apply to the department and with specific reporting on the department’s data center consolidation plan. The Secretary offered no comments on our recommendations, but noted that Commerce plans to address GAO’s finding on the department’s consolidation master program schedule in the next version of its consolidation plan. Commerce’s written comments are provided in appendix V.

In written comments, Defense’s CIO partially concurred with one of our recommendations and concurred with the second. Specifically, regarding our recommendation that the department complete the missing elements from its data center inventory and consolidation plan, the CIO cited the importance of completing consolidation metrics and noted that many of the department’s centers and buildings are not equipped to meter energy usage and that using incomplete estimates of such usage would result in inaccurate extrapolations of cost savings. However, OMB addressed such concerns in its guidance on the FDCCI, noting alternative means by which agencies could develop energy utilization estimates. OMB further recognized that these estimates may need time to become more accurate. As such, we believe our recommendation is reasonable and appropriate. Defense’s written comments are provided in appendix VI.

In written comments, Education’s CIO concurred with one recommendation and outlined plans to address the second. The CIO noted the department’s plans to complete, to the extent practicable, the missing information in Education’s data center inventory and consolidation plan. The CIO also cited the department’s intent, when updating its annual consolidation plan, to consider relevant consolidation challenges and lessons learned. Education’s written comments are provided in appendix VII.

In written comments, Energy’s Director of the Corporate IT Project Management Office agreed with our assessment of Energy’s data center consolidation plan and offered no comments on our recommendations. The Director cited a series of planned actions by the department intended to gather missing information in Energy’s data center inventory, update the department’s consolidation plan, and document its data center management best practices. Energy’s written comments are provided in appendix VIII.

In written comments, HHS’ Assistant Secretary for Legislation stated that the draft accurately depicts the HHS data center consolidation plan as it was delivered to OMB in August 2010 and notes that the agency has continued to improve its inventory and make progress on its data center consolidation goals since that time. Further, the Assistant Secretary outlined a series of actions planned by the department to complete HHS’ data center inventory and consolidation plan. The department did not offer comments on our recommendations. HHS’ written comments are provided in appendix IX.

In written comments, DHS’s Director of the Departmental GAO/OIG Liaison Office concurred with our recommendations. Further, the Director outlined the department’s planned actions to complete the missing information from its data center inventory and noted that the department is working to share its consolidation lessons learned with, among others, the Federal Chief Information Officers Council. DHS’s written comments are provided in appendix X.

In written comments, Interior’s Assistant Secretary for Policy, Management, and Budget concurred with our findings and recommendations. The Assistant Secretary also noted that the department is continuing to refine its data center inventory and consolidation goals. Interior’s written comments are provided in appendix XI.

In comments provided via e-mail, the Justice Audit Liaison concurred with our recommendations. Justice also provided technical comments, which we incorporated as appropriate.

In written comments, Labor’s Assistant Secretary for Administration and Management stated that, after carefully reviewing the draft report, the department did not have any comments to contribute. Labor’s written comments are provided in appendix XII.

In written comments, State’s Chief Financial Officer concurred with our recommendations and outlined a series of actions planned by the department to complete State’s data center inventory and consolidation plan. State’s written comments are provided in appendix XIII.

In comments provided via e-mail, Transportation’s Deputy Director of Audit Relations stated that the department had no comments on the report and agreed to consider our recommendations.

In written comments, Treasury’s Deputy Assistant Secretary for Information Systems and Chief Information Officer did not provide comments on our recommendations, but noted that Treasury has started its annual data center inventory collection, which will address missing data elements, and that Treasury intends to collect and leverage data center consolidation challenges and lessons learned when updating the department’s consolidation plans. Treasury’s written comments are provided in appendix XIV.

In written comments, VA’s Chief of Staff generally agreed with the findings and concurred with our recommendations. Further, the Chief of Staff noted planned actions to complete missing information from the department’s consolidation plan and to supplement updates to the plan with narrative responses on consolidation challenges and lessons learned. The Chief of Staff also noted that the department is continuing to refine its data center inventory, consolidation goals, and consolidation timeline. VA’s written comments are provided in appendix XV.

In written comments, EPA’s Assistant Administrator and Chief Information Officer did not agree or disagree with our recommendations, but did offer clarification on its plans to fulfill our recommendations. Specifically, in relation to our recommendation to complete missing elements of the agency’s consolidation plan, the Assistant Administrator clarified that the majority of EPA server rooms are located within leased space managed by GSA. As such, EPA will be able to estimate operating electrical use for these rooms, but the cost-benefit analysis will not reflect a reduction in real estate costs or electricity consumption because such reductions would not result in cost savings to EPA.

Further, the CIO asserted that we had mischaracterized the agency’s overall plan for consolidation by stating that EPA does not plan to further consolidate its four data centers and in our description of actions the agency plans to take within those four centers. Specifically, EPA stated that we did not provide adequate detail about EPA’s existing infrastructure or the services that will be provided by these four existing centers. However, EPA’s data center consolidation plan states that the agency had four data centers as of the end of fiscal year 2010 and the agency plans to have four data centers at the end of fiscal year 2014. As such, we maintain that our description of EPA’s broad consolidation goals is factual. EPA’s written comments are provided in appendix XVI.

In written comments, GSA’s Administrator agreed with both our findings and our recommendations and stated that GSA would take actions commensurate with our recommendations. GSA’s written comments are provided in appendix XVII.

In written comments, NASA’s CIO concurred with our recommendations. Further, the CIO cited ongoing work by NASA to complete the missing information in the agency’s data center inventory and consolidation plan. Additionally, the CIO noted the agency’s plans to consider consolidation challenges and lessons learned when updating consolidation plans. NASA’s written comments are provided in appendix XVIII.

In comments provided via e-mail, NSF’s Acting CIO did not provide comments on our recommendations, but noted NSF’s planned actions to complete the missing information from the agency’s consolidation plan.

In written comments, NRC’s Deputy Executive Director for Corporate Management within the Office of the Executive Director for Operations stated that the agency had no comments. NRC’s written comments are provided in appendix XIX.

In comments provided via e-mail, OPM’s Deputy CIO concurred with our recommendations. The Deputy CIO noted that since OPM does not plan to consolidate further than its one data center, the agency’s consolidation focus will be to complete its asset inventory and explore ways to operate more efficiently.

In comments provided via e-mail, the SBA Program Manager for the Office of Congressional and Legislative Affairs concurred with our recommendations.

In written comments, SSA’s Deputy Chief of Staff disagreed with one recommendation and agreed with the second. Specifically, regarding our recommendation that the agency complete the missing elements from its data center inventory and consolidation plan, the Deputy Chief of Staff disagreed with our assessment of SSA’s asset inventory and consolidation plan, stating that SSA responded to OMB’s directive in a timely and satisfactory manner. The Deputy Chief of Staff further noted that because SSA does not plan to consolidate its two existing data centers, the plan elements we noted as missing were not applicable to SSA’s circumstances.

However, in its asset inventory, SSA does not provide one of the OMB-specified consolidation approaches, which includes an option of “not applicable,” for any of the major and nonmajor systems in SSA’s data centers. While we acknowledge that SSA does not plan to consolidate from its two physical locations, OMB still required agencies to provide a consolidation approach for every identified system. Further, in a written response to OMB questions about SSA’s consolidation plan, SSA acknowledged that the agency planned to virtualize systems within one of its two locations. In light of this planned work, it is reasonable to assume that an agency would complete the important governance-related key plan elements we identified as missing, such as a master program schedule, a cost- benefit analysis, and a risk management plan. In its guidance on the FDCCI, OMB echoed this importance, noting that an agency’s governance framework needs to provide specific details about the oversight and internal mechanics that will measure and manage performance and risk of the consolidation implementation. As such, we believe our recommendation is reasonable and appropriate. SSA’s written comments are provided in appendix XX.

In comments provided via e-mail, the liaison from USAID’s Office of the Chief Financial Officer concurred with our recommendations.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to interested congressional committees; the Director of OMB; the secretaries and agency heads of the departments and agencies addressed in this report; and other interested parties. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any questions on the matters discussed in this report, please contact me at (202) 512-6253 or willemssenj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix XXI.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to (1) assess whether agency consolidation documents include adequate detail, such as performance measures and milestones, for agencies to consolidate their centers; (2) identify the key challenges reported by agencies in consolidating centers; and (3) evaluate whether lessons learned during state government consolidation efforts could be leveraged to mitigate challenges at the federal level.

For this governmentwide review, we assessed the 24 departments and agencies (agencies) that were identified by the Office of Management and Budget (OMB) and the Federal Chief Information Officer (CIO) to be included in the Federal Data Center Consolidation Initiative (FDCCI). Table 6 lists these agencies.

To evaluate the agencies’ data center inventories and consolidation plans, we reviewed OMB’s guidance and identified key required elements for each. We compared agency consolidation inventories and plans to OMB’s required elements, and identified gaps and missing elements. We rated each element as “Yes” if the agency provides complete information; “Partial” if the agency provides some, but not all, of the information; and “No” if the agency did not provide the information. We followed up with agencies to clarify our initial findings and to determine why parts of the inventories and plans were incomplete or missing. We assessed the reliability of the data agencies provided in their data center inventories and plans. Specifically, we interviewed agency officials to determine how the data in the inventories and plans had been collected and their processes for ensuring the reliability of the data contained in these inventories. We reviewed the inventories and plans for omissions, outliers, and typographic mistakes. We compared inventory summary data contained in the consolidation plans to inventories and noted any inconsistencies. In doing so, we found multiple gaps in agency-provided data. We also found that almost half of the agencies had not taken steps to verify their inventory data. We have reported on these limitations in the body of this report.

To identify the key challenges encountered by agencies in consolidating data centers, we analyzed available literature on data center consolidation challenges and interviewed agency officials to determine what challenges to consolidation had been encountered. We then categorized the agency-reported challenges to determine ones that were encountered most often.

To evaluate whether lessons learned during state government consolidation efforts could be leveraged to mitigate challenges at the federal level, we conducted a literature search for information on state experiences in data center consolidation and interviewed the National Association of State Chief Information Officers regarding states’ experiences with data center consolidation. These sources identified 20 states that reported challenges or lessons learned from their data center consolidation initiatives. We sought clarification on challenges and lessons learned through e-mail and interviews with state officials. We compared states’ challenges and lessons learned to the challenges facing federal agencies in order to identify which lessons learned could be applied to federal consolidation efforts.

We conducted our work at multiple agencies’ headquarters in the Washington, D.C., metropolitan area. We conducted this performance audit from August 2010 to July 2011, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Assessment of Agencies’ Completion of Consolidation Planning Elements, Arranged by Key Element

Table 7 identifies the agencies that provide complete, partial, or no information for each key element of its data center consolidation plan.

Appendix III: Assessment of Agencies’ Completion of Key Consolidation Planning Elements, Arranged by Agency

As part of its data center consolidation initiative, OMB required 24 federal departments and agencies to submit a data center inventory and a data center consolidation plan. Key elements of the inventory were to include, for each data center, information on IT hardware, IT software, facilities/energy/storage, and geographic location. Key elements of the plan were to include information on quantitative goals, qualitative impacts, consolidation approach, consolidation scope, timeline, performance metrics, master schedule, cost-benefit analysis, risk management, and consideration of a communications plan.

For each of the agencies, the following sections provide a brief summary of the agencies’ goal for reducing the number of data centers, and an assessment of the completeness of their inventories and plans.

The following information describes the key that we used in tables 8 through 30 to convey the results of our assessment of the agencies’ compliance with OMB’s requirements for the FDCCI. ● The agency provides complete information for this element. ◐ The agency provides some, but not all, aspects of the element. ○ The agency does not provide information for this element.

Department of Agriculture

Agriculture plans to consolidate from 46 data centers to 7 by fiscal year 2015. However, the agency’s asset inventory and consolidation plan are not complete. In its asset inventory, the department provides complete information for 1 element and partial information for the remaining 3 elements. Additionally, in its consolidation plan, Agriculture provides complete information for 8 of the 10 elements, partial information for 1 element, and does not provide information for the remaining element. Table 8 provides our assessment of Agriculture’s compliance with OMB’s requirements.

Commerce plans to consolidate from 41 data centers to 23 by fiscal year 2015. However, the agency’s asset inventory and consolidation plan are not complete. In its inventory, the department provides complete information for 3 of the 4 key elements and partial information for the remaining element. Additionally, in its consolidation plan, Commerce provides complete information for 8 of the 10 elements evaluated, provides partial information for 1 element, and does not provide information for the remaining element. Table 9 provides our assessment of Commerce’s compliance with OMB’s requirements.

Defense plans to consolidate from 772 data centers to 532 by fiscal year 2013. However, the agency’s asset inventory and consolidation plan are not complete. In its asset inventory, Defense provides only partial information for all 4 key elements. Additionally, in its consolidation plan, Defense provides complete information for 3 of the 10 elements evaluated, provides partial information for 5 elements, and does not provide information for the remaining 2 elements. A Defense official explained that because they are a decentralized agency and are fighting multiple wars, it was difficult to meet OMB’s extremely short deadlines. The official also noted that OMB’s changing templates and definitions made it more difficult to compile the needed information. Table 10 provides our assessment of Defense’s compliance with OMB’s requirements.

Education does not have plans to consolidate any of its three data centers before fiscal year 2015. Rather, the agency plans to increase server virtualization within in its centers. However, Education’s asset inventory and consolidation plan are not complete. In its asset inventory, the agency provides complete information for 3 key elements and provides partial information for the remaining element. Additionally, in its consolidation plan, Education provides complete information for 4 of the 10 elements evaluated, provides partial information for 3 elements, and does not provide information for the remaining 3 elements. Education officials stated that they did not provide selected plan elements because they are not applicable given the agency’s focus on virtualization rather than consolidation. Table 11 provides our assessment of Education’s compliance with OMB’s requirements.

Energy plans to consolidate from 31 data centers to 25 by fiscal year 2015. However, the agency’s asset inventory and consolidation plan are not complete. In its asset inventory, Energy reports that although it began identifying contractor-operated data centers, this initiative was not completed in time to be included in the inventory. Additionally, in its consolidation plan, Energy provides complete information for 3 of the 10 elements evaluated, provides partial information for 2 elements, and does not provide information for the remaining 5 elements. Table 12 provides our assessment of Energy’s compliance with OMB’s requirements.

HHS plans to consolidate from 185 data centers to 131 by fiscal year 2015. However, the agency’s asset inventory and consolidation plan are not complete. In its asset inventory, the department provides partial information for all 4 key elements. Additionally, in its consolidation plan, HHS provides complete information for 6 of the 10 elements evaluated, provides partial information for 1 element, and does not provide information for the remaining 3 elements. Agency officials stated that they are working to complete the elements that are missing or incomplete. Table 13 provides our assessment of HHS’ compliance with OMB’s requirements.

DHS plans to consolidate from 43 data centers to 2 by fiscal year 2014. However, the agency’s asset inventory and consolidation plan are not complete. In its asset inventory, DHS provides partial information for all 4 key elements. Additionally, in its consolidation plan, the department provides complete information for 8 of the 10 elements evaluated, provides partial information for 1 element, and does not provide information for the remaining element. Table 14 provides our assessment of DHS’s compliance with OMB’s requirements.

The Department of Housing and Urban Development did not submit a data center inventory or consolidation plan. Instead, it submitted a letter that asserts that the department does not own any data centers and has no arrangements to take ownership of any data centers at the end of any contracts.

Department of the Interior

Interior plans to consolidate from 95 data centers to 5 by fiscal year 2015. However, the agency’s asset inventory and consolidation plan are not complete. In its asset inventory, Interior provides complete information for 1 key element and partial information for the remaining 3 elements. In its consolidation plan, Interior provides complete information for 8 of the 10 elements evaluated, and provides partial information for 2 elements. Interior officials stated that they are working to complete the elements that are missing or incomplete. Table 15 provides our assessment of Interior’s compliance with OMB’s requirements.

Justice plans to consolidate from 65 data centers to 50 by fiscal year 2015. However, the agency’s asset inventory and consolidation plan are not complete. In its asset inventory, Justice provides complete information for 1 key element and partial information for the remaining 3 elements. Additionally, in its consolidation plan, Justice provides complete information for 5 of the 10 elements evaluated, provides partial information for 2 elements, and does not provide information for the remaining 3 elements. Table 16 provides our assessment of Justice’s compliance with OMB’s requirements.

Labor plans to consolidate from 20 data centers to 18 by fiscal year 2015. However, the agency’s asset inventory and consolidation plan are not complete. In its asset inventory, Labor provides complete information for 1 key element and provides partial information for the remaining 3 elements. Additionally, in its consolidation plan, Labor provides complete information for 7 of the 10 elements evaluated, provides partial information for 1 element, and does not provide information for the remaining 2 elements. Table 17 provides our assessment of Labor’s compliance with OMB’s requirements.

State plans to consolidate from 13 data centers to 6 by fiscal year 2015. However, the agency’s asset inventory and consolidation plan are not complete. In its asset inventory, State provides complete information for 3 of the key elements and provides partial information for the remaining element. Additionally, in its consolidation plan, State provides complete information for 6 of the 10 elements evaluated, provides partial information for 3 elements, and does not provide information for the remaining element. An agency official stated that they have a master program schedule and performance metrics, but acknowledged that they did not provide them to OMB as part of their consolidation plans. Table 18 provides our assessment of State’s compliance with OMB’s requirements.

Transportation plans to consolidate from 35 data centers to 31 by fiscal year 2015. However, the agency’s asset inventory and consolidation plan are not complete. In its inventory, Transportation provides partial information for all 4 key elements, noting that in some instances, data center owners did not provide the requested information. Additionally, in its consolidation plan, Transportation provides complete information for 9 of the 10 elements evaluated and does not provide information for the remaining element. Table 19 provides our assessment of Transportation’s compliance with OMB’s requirements.

Treasury plans to consolidate from 42 data centers to 29 by fiscal year 2015. However, the agency’s asset inventory and consolidation plan are not complete. In its inventory, Treasury provides complete information for 1 key element and provides partial information for the remaining 3 elements. Additionally, in its consolidation plan, Treasury provides complete information for 5 of the 10 elements evaluated, provides partial information for 1 element, and does not provide information for the remaining 4 elements. An agency official stated that the agency is working to complete the missing or incomplete items. Table 20 provides our assessment of Treasury’s compliance with OMB’s requirements.

VA plans to consolidate 87 data centers into 4 by fiscal year 2015. However, the agency’s asset inventory and consolidation plan are not complete. In its inventory, VA provides partial information for all 4 key elements. Additionally, in its consolidation plan, VA provides complete information for 6 of the 10 elements evaluated, provides partial information for 2 elements, and does not provide information for the remaining 2 elements. Table 21 provides our assessment of VA’s compliance with OMB’s requirements.

EPA does not plan to further consolidate its four primary data centers. Instead, the agency plans to focus its consolidation efforts on achieving efficiencies via virtualization within those four centers. However, the agency’s asset inventory and consolidation plan are not complete. In its inventory, the agency provides complete information for 3 of the key elements and provides partial information for the remaining element. Additionally, in its consolidation plan, EPA provides complete information for 5 of the 10 elements evaluated, provides partial information for 1 element, and does not provide information for the remaining 4 elements. Table 22 provides our assessment of EPA’s compliance with OMB’s requirements.

GSA plans to consolidate from 15 data centers to 3 by fiscal year 2015. However, the agency’s asset inventory and consolidation plan are not complete. In its inventory, the agency provides complete information for 2 of the key elements and provides partial information for the remaining 2 elements. Additionally, in its consolidation plan, GSA provides complete information for 8 of the 10 elements evaluated, provides partial information for 1 element, and does not provide information for the remaining element. Table 23 provides our assessment of GSA’s compliance with OMB’s requirements.

NASA plans to consolidate from 79 data centers to 57 by fiscal year 2015. However, the agency’s asset inventory and consolidation plan are not complete. In its inventory, the agency provides partial information for 3 of the key elements and does not provide information for the remaining element. Additionally, in its consolidation plan, NASA provides complete information for 4 of the 10 elements evaluated, provides partial information for 1 element, and does not provide information for the remaining 5 elements. Table 24 provides our assessment of NASA’s compliance with OMB’s requirements.

NSF owns and operates one data center and utilizes one commercial data center. The agency aims to transition all operations to one commercial data center by fiscal year 2014. The agency’s asset inventory is complete, but its consolidation plan is not. Specifically, NSF provides complete information for 7 of the 10 elements evaluated, and does not provide information for the remaining 3 elements. Agency officials stated that they have a master schedule and risk management plan, but acknowledged that they did not provide this information to OMB as part of their consolidation plan. Table 25 provides our assessment of NSF’s compliance with OMB’s requirements.

NRC plans to consolidate from three existing data centers into one new center by fiscal year 2013. However, the agency’s asset inventory and consolidation plan are not complete. In its inventory, the agency provides complete information for 3 of the key elements and provides partial information for the remaining element. Additionally, in its consolidation plan, NRC provides complete information for 4 of the 10 elements evaluated, provides partial information for 1 element, and does not provide information for the remaining 5 elements. Table 26 provides our assessment of NRC’s compliance with OMB’s requirements.

OPM does not plan to further consolidate its one data center. Instead, the agency plans to continue to examine and execute ways to improve the efficiency of its IT operations, such as through virtualization. However, the agency’s asset inventory and consolidation plan are not complete. In its inventory, the agency provides complete information for 1 key element and partial information for the remaining 3 elements. Additionally, in its consolidation plan, OPM provides complete information for 5 of the 10 elements evaluated, provides partial information for 1 element, and does not provide information for the remaining 4 elements. Table 27 provides our assessment of OPM’s compliance with OMB’s requirements.

SBA plans to reduce its number of data centers from four to two by fiscal year 2015. However, the agency’s asset inventory and consolidation plan are not complete. In its inventory, the agency provides complete information for 1 key element and partial information for the remaining 3 elements. Additionally, in its consolidation plan, SBA provides complete information for 5 of the 10 elements evaluated, partial information for 1 element, and does not provide information for the remaining 4 elements. Table 28 provides our assessment of SBA’s compliance with OMB’s requirements.

SSA does not plan to further consolidate its two data centers. In line with the goals of the FDCCI, the agency plans to improve the efficiency, performance, and stability of its IT infrastructure by reducing the number of its remote operations control centers. However, the agency’s asset inventory and consolidation plan are not complete. In its inventory, the agency provides partial information for all 4 key elements. Additionally, in its consolidation plan, SSA provides complete information for 4 of the 10 elements evaluated, partial information for 1 element, and does not provide information for the remaining 5 elements. Table 29 provides our assessment of SSA’s compliance with OMB’s requirements.

USAID plans to consolidate from two data centers into one by fiscal year 2015. However, the agency’s asset inventory and consolidation plan are not complete. In its inventory, the agency provides complete information for 1 key element and partial information for the remaining 3 elements. Additionally, in its consolidation plan, USAID provides complete information for 5 of the 10 elements evaluated, provides partial information for 1 element, and does not provide information for the remaining 4 elements. Table 30 provides our assessment of USAID’s compliance with OMB’s requirements.

Appendix IV: OMB-Defined Approaches for Consolidating Data Centers

In its guidance on data center consolidations, OMB identified four approaches for agencies to consider while evaluating the feasibility of consolidating the individual systems found within each data center. OMB directed agencies to specify which of these approaches was to be utilized for each data center. The four approaches are as follows:  Decommissioning: the system is no longer in use or it is redundant and will be decommissioned.  Consolidation: the system will be consolidated onto a shared infrastructure with other similar systems.  Cloud computing: the system will be migrated to or replaced by Internet-based services and resources.  Virtualization: the system will be migrated to a virtual machine environment.

In response to OMB’s guidance, agencies reported that they will pursue a variety of consolidation approaches. Agency-specific examples of how these approaches will be employed are provided below.

Decommissioning

Agencies may choose to decommission their underutilized physical servers as a part of their data center consolidation plans. For example, EPA plans to decommission more than 900 physical servers by 2015. Also, Labor plans to decommission unused servers and storage hardware and replace inefficient hardware with “green IT” hardware. Further, GSA plans to identify and decommission inefficient and underutilized legacy servers and equipment.

Agencies can also choose to decommission an entire data center by moving to an outsourced data center or reducing the number of physical assets. For example, as part of its data center consolidation initiative, NSF plans to decommission its single data center by fiscal year 2014 and to move to a commercial facility. Transportation plans to decommission data centers that spread out across multiple buildings and reduce the department’s number of data centers by approximately 25 percent by the close of fiscal year 2015.

Consolidation

Consolidation is a means of combining workload onto fewer computers or concentrating data processing into fewer physical facilities. Physically moving data processing equipment from multiple locations to a lesser number of locations can assist agencies in reaching consolidation goals, such as reducing the cost of data center hardware, software, and operations, in addition to real estate and energy costs. For example, DHS has 43 principal data centers, all of which will be moved into one of two enterprise data centers by the end of fiscal year 2014. In addition, NASA plans to consolidate from 79 data centers to 57 by fiscal year 2015.

Cloud computing

Cloud computing is an emerging form of computing that relies on Internet- based services and resources to provide computing services to customers, while freeing them from the burden and costs of maintaining the underlying infrastructure. This approach is a form of delivering IT services that takes advantage of several broad evolutionary trends, including the use of virtualization; the decreased cost and increased speed of networked communications, such as the Internet; and overall increases in computing power. Examples of cloud computing include Web-based e-mail applications and common business applications that are accessed online through a browser instead of through a local computer. Several agencies are considering both cloud computing and virtualization as a means of achieving their consolidation goals. For example, SBA has plans to migrate commodity computing services such as Web hosting and messaging to cloud solutions. We have recently reported on challenges associated with the implementation of cloud computing.

Virtualization

Virtual machines can be stored as files, making it possible to save a virtual machine and move it from one physical server to another. Virtualization is often used as part of cloud computing. For example, one Defense component reports that 45 percent of all server operating environments supporting customer workload in its data centers have been virtualized. State plans to reduce its environmental impact by hosting 70 percent of the department’s servers on virtual infrastructure by 2015. Also, NRC has virtualized 41 Windows-based applications and has identified 50 additional applications to be virtualized by 2013.

Appendix V: Comments from the Department of Commerce

Appendix VI: Comments from the Department of Defense

Appendix VII: Comments from the Department of Education

Appendix VIII: Comments from the Department of Energy

Appendix IX: Comments from the Department of Health and Human Services

Appendix X: Comments from the Department of Homeland Security

Appendix XI: Comments from the Department of the Interior

Appendix XII: Comments from the Department of Labor

Appendix XIII: Comments from the Department of State

Appendix XIV: Comments from the Department of the Treasury

Appendix XV: Comments from the Department of Veterans Affairs

Appendix XVI: Comments from the Environmental Protection Agency

Appendix XVII: Comments from the General Services Administration

Appendix XVIII: Comments from the National Aeronautics and Space Administration

Appendix XIX: Comments from the Nuclear Regulatory Commission

Appendix XX: Comments from the Social Security Administration

Appendix XXI: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Colleen Phillips (Assistant Director), Neil Doherty, Rebecca Eyler, Nancy Glover, Dave Hinchman, Linda Kochersberger, and Jessica Waselkow made key contributions to this report.