U.S. Department of Energy 

Office of Inspector General 

Office of Audit Services 

Audit Report 

Management of the Department's 

Data Centers at Contractor Sites 

DOE/IG-0803 October 2008 
Department of Energy 

Washington, DC 20585 

October 30, 2008 

MEMORANDUM FOR ?'HE SECRETARY 

FROM: 

Inspector Gcneral 

SUBJECT: INFORMATION: Audit Report on "Management of Data 

Centers at Contractor Sites" 

BACKGROUND 

The Department of Energy (Department) and its facility contractors maintain numerous 

data centers across the complex that varies in both size and complexity. These centers, 

which house data and voice hardware and software, and a variety of network equipment, 

typically require specialized security, power, and cooling capabilities. Thc centers 

provide common information technology (IT) services in support of the Department's 

most critical functions, including scientific research and national security activities. The 

cost to acquire data center space and the hardware, software and services necessary to 

sustain operations are signiticant - amounting to about $33 1 million in Fiscal Year 2007. 

Recent reviews by the Office of Inspector General (OIG) have identified a number of 

areas in which the Department could improve the efficiency of its vast IT efforts. For 

cxaniple, a series of reports revealed that significant operating savings were available by 

aggregating demand for IT goods and services. A number of Federal and private sector 

organizations have also realized meaningful savings by consolidating data centers, 

centralizing delivery of common services, and employing more efficient network and 

computing technologies. Based on these findings, we initiated this audit to determine 

whether the Department had effectively managed its data centers at contractor sites. 

-- RESULTS OF AUDIT 

'The Department had not always taken advantage of opportunities to improve the 

eficiency of its contractor data centers. In particular, we evaluated operations at six data 

centers and concluded that: 

Despite opportunities to consolidate requirements and physical operations, the six 

sites maintained as many as 140 independent data centers. These data centers 

duplicated many of the functions or services provided by other co-located centers; 

In many instances, multiple data centers located at the same site separately 

provided common services such as email, data storage and libraries - services that 

were not unique and which appeared to readily lend themselves to consolidation; 

and, 
Four of six sites made only limited use of more efiicicnt hardware technologies 

that conserve energy and reduce operational costs. 

We found that neither the National Nuclear Security Administration (NNSA) nor the 

Department's program offices had required contractors to focus on reducing overall data 

center costs, including those related to energy use. Specifically, officials had not 

provided guidance or adequately colnmunicated best practices to contractors and field 

sites regarding opportunities to consolidate data centers and improve the efficiency of IT 

hardware and services. Such operating enhancements would help to ensure that taxpayerprovided funds are used economically and efficiently. 

As outlined in the body of the attached report, our audit disclosed a number of areas in 

which improvements are possible. To put this in some perspective, at just the six sites 

includcd in our review, we estimated that the Depart~nent could save over $2.3 million 

annually through the use of more efficient hardware technologies that enable the 

consolidation of servers. Such actions would also make data centers more energy 

eflicient, consistent with Departmental goals. 

We noted, however, scveral signs of progress in this arena. For example, the 

Department's Office of Energy Efficiency and Renewable Energy (EERE) has partnered 

with industry and government agencies to improve energy efficiency in the nation's data 

centers. In an effort to reduce energy consumption in the Nation's data centers by 10 

percent by 201 1, EERE has developed assessment tools and providcd information and 

training on data center energy efficiency to the Department, other Federal, and private 

sector organizations. Lawrence Berkeley National Laboratory has also undertaken efforts 

to improve the efficiency of data ccnters by designing systems to reduce energy 

consumption. At Headquarters, the Office of the Chief Infortnation Officer has 

consolidated common services and implemented application hosting for most program 

elements and a similar initiative is being considered for Federal data centers across the 

complex. Further, several Department of Energy contractor-operated sites have taken 

steps to consolidate services, including e-mail. While these efforts are commendable, a 

number of opportunities remain to address redundant contractor data centers and services, 

and to consolidate activities. Accordingly, we have made recommendations designed to 

help the Department optimize the operation of its contractor-operated data centers across 

the complex. 

MANAGEMENT REACTION 

NNSA management concurred with our recommendations and agreed to take corrective 

action. Consolidated comments provided by other Department elements indicated that 

they also concurred with each of our recommendations and planned certain corrective 

actions. However, management expressed disagreemcnt with several of our conclusions. 

Management's commcnts are discussed in greater detail in Appendix 3. 

Attachment 
cc: Acting Deputy Secretary 

Administrator, National Nuclear Security Administration 

Llnder Secretary of Energy 

Under Secretary for Science 

Chief Information Officer 

Chief of Staff 
REPORT ON MANAGEMENT OF THE DEPARTMENT'S DATA 

CENTERS AT CONTRACTOR SITES 

TABLE OF 

CONTENTS 

Management of Data Centers at Contractor Sites 

Lletails of Finding ........................................................................................................ 1 

Recommendations and Comments. .................................................................................. .6 

Management Reaction ...................................................................................................... 7 

Appendices 

1. Ob.jective, Scope, and Methodology .......................................................................... 10 

2. Prior Reports .............................................................................................................. 12 

3. Management Comments.. ..................................................................... 14 
Manaqement of Contractor Data Centers 

Contractor Data The Department of Energy (Department) had not always 

Centers taken advantage of opportunities to improve the operational 

and energy efficiency of its data centers at contractor sites. 

Despite opportunities to consolidate requirements and 

physical operations, sites, and programs within sites, 

independently maintained numerous data centers that, in 

many respects, duplicated functions or services provided by 

other co-located centers. In many instances, multiple data 

centers located at the same site separately provided 

common services such as email, data storage and libraries - 

services that were not unique and which readily lend 

themselves to consolidation. Sites also made only limited 

use of more efficient hardware technologies that conserve 

energy and reduce operational costs, practices that could 

ultimate1 y permit the consolidation of data centers. 

Consolidation 

Many of the entities at the six field sites we visited 

individually maintained data centers despite opportunities 

to consolidate and centrally manage these operations. At 

Argonne National Laboratory (Argonne) and Lawrence 

Livermore National Laboratory (Livermore), for example, 

several programs maintained their own center, with little or 

no coordination of data center services between the 

programs. Officials at Los Alamos National Laboratory 

(Los Alamos) informed us that there may be as many as 

100 unclassified data centers on the site, most of' which 

they believed could be consolidated. In addition, we 

identified almost 40 data centers at the other five sites we 

visited. Annual costs to operate and maintain just 14 of 

these data centers exceeded $1 5 million. Officials told us 

that they did not maintain or have cognizance over costs for 

the other 100 plus data centers because the costs were 

either built into the overhead cost for the building or the 

site did not individually charge for each center. It should 

be noted that, subsequent to our review at Livermore, 

officials told us that Liverrnore has begun to actively 

pursue consolidating some of their institutional services 

into their new institutional data center. 

Many of the sites we visited had also not taken action to 

utilize available excess data center space. Four of the data 

centers we visited had excess space and had offered the 

space to other organizations. However. organizations 

within those sites chose not to take advantage of the 

Page 1 Details of Finding 
opportunity for consolidation and continued to maintain 

their own functionally duplicate data center. An official at 

Argonne stated that each program maintained its own 

center because funding for the data centers is individually 

received by each program. Nonetheless, the official 

observed that there may be opportunities for consolidation. 

Despite opportunities to combine services, sites operating 

multiple data centers often housed servers providing 

common services that lend themselves to consolidation. 

Many sites had consolidated enterprise, or corporate 

systems. such as payroll and other financial services within 

a single data center. However, marly of these same sites 

were still operating separate common services, such as 

email, data storage. and data libraries at various other data 

centers on site. For example, Argonne had five scientific 

data centers that ran similar data storage and libraries for 

each center, services which may be capable of being 

consolidated and housed in a single center. 

Application Hosting and Virtualization 

Sites made only limited use of more efficient technologies, 

such as application hosting or virtualization environments. 

For example, only one site reviewed utilized an application 

hosting environment. Application hosting consists of a 

central or host organization providing full information 

technology (IT) services, including housing, securing, 

operating, and maintaining the servers or other devices for 

client organizations. This environment eliminates or 

decreases the need for each organization to individually 

operate and support its own servers, thereby reducing 

overall costs. We found that all six sites we visited made 

some use of a data center housing environment, in which 

multiple organizations individually operate and maintain 

their own servers and applications in a centralized 

environment. However, while this practice reduced the 

need for individual data centers, each organization still 

utilized its own resources to operate and maintain its 

equipment - an inefficient and at least partially duplicative 

support model. 

Additionally, only three of the sites we visited, Los 

Alamos, Lawrence Berkeley Laboratory, and Sandia 

National Laboratory (Sandia), had taken limited advantage 

of existing technologies, such as virtualization, to 

consolidate their file servers. Virtualization allows 

-- 

Page 2 Details of Finding 
multiple operating systems to run concurrently on a single 

server, thus maximizing the use of the machine, minimizing 

the need for multiple servers for each operating system, and 

reducing energy consumption. At Los Alamos, even 

though virtualization *as used only on a limited basis. 

officials estimated that their efforts would achieve annual 

savings of $1.6 million. These same officials noted that 

significant underutilization of hardware existed prior to the 

move to virtualization. They also stated that virtualization 

is currently only being applied to business and institutional 

services; however, they hope to expand the process in the 

future and anticipate that they will be able to further reduce 

the number of physical servers by 50 percent. Officials at 

Sandia stated they had over 100 virtual machines, just a 

partial consolidation of the site's servers, resulting in annual 

savings of more than $700,000. 

Data Center Guidance Program officials and the National Nuclear Security 

and Management Administration (NNSA) had not required contractors to 

focus on reducing data center costs and energy use. Also, 

best practices and opportunities to consolidate and improve 

the efficiency of IT hardware and services had not been 

communicated to contractors and field sites. Sites had not 

monitored server utilization to determine whether excess 

capacity existed or whether efficiencies could be achieved 

through virtualization of file servers. 

Program Guidance 

Despite Federal requirements that agencies maximize the 

value of their IT resources. NNSA and the Department's 

program offices had not provided guidance to contractors 

and field sites requiring them to examine opportunities for 

more cost-effective management of their data centers. 

Although the Department had been successful in 

consolidating operations and reducing data center costs in 

certain areas, officials had not taken action to require 

contractors to minimize costs. The Department had 

undertaken an effort to, among other things, consolidate its 

Federal data centers and reduce the number of file servers 

by creating a Federal application hosting environment as a 

part of the Department Common Operating Environment 

initiative at Headquarters. Officials from the Office of the 

Chief Information Officer estimated that these 

consolidation efforts would permit them to save or avoid 

costs of about $1 5 million in just the first year. Also, both 

Federal and certain contractor 

Page 3 Details of Finding 
organizations have realized savings as a result of several 

relatively small, independent initiatives to reduce the 

number of servers through virtualization. Despite these 

successes, these best practices had not been comn~unicated 

to contractors at Geld sites nor had the program offices 

directed thein to determine whether they could achieve 

similar cost efficiencies through data center and file server 

consolidations. 

Site officials we spoke with identified several reasons for 

not pursuing data center consolidation. Specifically, some 

sites indicated that their data centers had power and cooling 

problems. so they did not want to consolidate operations 

and possibly aggravate the situation. However, the use of 

more efficient information technologies, such as 

virtualization, can significantly reduce the number of 

servers necessary to run applications, resulting in 

corresponding savings in power and cooling. Some sites 

indicated that their funding was constrained and without 

specific direction, they wanted others to take the lead on 

using innovative approaches, such as virtualization. 

However, virtualization and application hosting are proven 

technologies that, as demonstrated by both Federal and 

private sector experience, result in near term savings. 

Monitoring Server Utilization 

Most of the sites we reviewed were not aware or were 

unable to quantify the benefits, including cost and energy 

savings. available from virtualization because they did not 

monitor server utilization. Even though they did not 

specifically monitor use of most systems, officials at lies 

Alamos told us that they believed that server hardware is 

underutilized, resulting in higher costs for data center 

maintenance and administration. Only one of the six sites 

visited, Livermore, monitored its server utilization at one of 

its data centers to ensure that excess capacity was not being 

acquired and existing computer resources were being 

maximized. 

Opportunities for Consolidation of data centers through practices such as 

Savings application hosting, virtualization and consolidation of 

conimon services could permit the Department's 

contractors to realize significant savings and achieve 

meaningful reductions in energy consumption. 

Page 4 Details of Finding 
Minimizing Costs 

Without improvements in Department guidance and 

contractor monitoring of server utilization, contractors and 

field sitcs will continue to spend more than necessary on 

data center operations. Specifically, sites reported they had 

2,433 enterprise servers' at five of the six sites we visited, 

many of which could be candidates for consolidation or 

virtualization. I wo field sites reported they were able to 

eliminate, on average, 60 percent of the servers they 

targeted for virtualization. For example, the Pacific 

Northwest National Laboratory (PNNL) reported that it 

realized savings of approximately $1 75,000 by 

consolidating only 1 13 servers. By extrapolating the 

savings that PNNL realized, which were the most 

conservative reported by various sites we polled. the 

Department may be able to realize savings of 

approximately $2.3 million just through virtualization and 

consolidation of servers at just five of the six sites we 

visited. 

Experience by Departmental organizations and results 

observed in industry also indicate that the projected savings 

by utilizing more efficient server technology are likely 

achievable. For example, Department organizations from 

which we obtained data indicated that their limited ef'rorts 

in this area have already resulted in $1 million in savings 

gained through consolidation efforts. A recent study on 

return on investments, conducted in July 2006 by industry 

experts, similarly reported that the Department saved over 

$2.3 million by performing very 1 imited virtualization of 

servers at just eight sites. These savings result from lower 

costs to acquire and maintain hardware, as many 

applications, common services andlor operating systems 

can run on the same server. Industry officials noted that 

virtuali~ation typically reduces a system administrator's 

workload by 60 to 90 percent. 

Consolidation and Reduced Energy Consumption 

While investment in server and data center consolidation 

may initially exceed normal operating costs, industry 

experts and contractor officials with successful 

consolidation experience agree that the payback period is 

' An enterprise server is defined as a computer system which perfornis an essential service for the 

organization, such as file servers, email servers, print servers, and web servers. It does not include research 

and development and other scientific computers. 

- 

Page 5 Details of Finding 
generally short and return on overall investment is 

substantial. Reductions in data center space can. in many 

instances, result in corresponding decreases in data center 

lease and maintenance costs, which are typically 10 to 100 

percent more expensive than normal office space. For 

example. the Defense Contract Management Agency 

recently consolidated 18 data centers into 2 through a 

virtualization process, reportedly saving the organization 

hundreds of thousands of dollars a year. 

Additional savings can also bc achieved through reduced 

energy consumption through server and data center 

consolidation. For example, an August 2007 study by the 

Environmental Protection Agency, relying on data provided 

by Hewlett Packard, determined that data center managers 

could cut energy consumption by as much as 40 percent if 

they adopted the most efficient technologies available. The 

study concluded that the government could save over $959 

million in five years just in reduced energy costs by 

adopting technologies to consolidate servers and enhance 

their ei'ficiency. Reducing energy consumption through 

server and data center consolidation will also help achieve 

the President's Order calling for executive agencies to 

improve energy efficiencies and reduce carbon emissions 

by three percent annually. 

RECOMMENDATIONS To help improve the efficiency of contractor-operated data 

centers. we recommend that thc Administrator, NNSA, and 

the Under Secretaries of Energy and for Science. in 

coordination with the Department and KNSA's Chief 

Information Officers: 

1. Share best practices and provide guidance and 

direction regarding data center consolidation 

through application hosting and server 

consolidation; 

We further recommend that the Administrator, NNSA, and 

the Under Secretaries of Energy and for Science: 

2. Require contractors and field sites to actively 

monitor server utilization and take action to more 

fully utilize existing servcr capacity: and, 

Page 6 Recommendations and Comments 
MANAGEMENT 

REACTION 

Page 7 

3. Require contractors and field sites to consolidate 

servers, data ccnters, and common services, where 

appropriate. 

NNSA management concurred with our rccomrnendations 

and noted that after considering any unique mission needs 

and best business practices, they will provide best practice 

information, guidance, or direction related to consolidation; 

monitor utilization; and encourage the appropriate 

consolidation of services. Department management 

generally concurred with the recommendations in the report 

and indicated that it supports the report's goal of improving 

data center efficiency and reducing the impact on the 

environment. l'he Department indicatcd that it planned to 

continue to share best practices through existing forums 

and would publish additional guidance if necessary, hut 

contemplated no additional action at this time. Although it 

did not detail specific actions to be taken, management 

indicated that it would take steps to require laboratories and 

sites to monitor and more fully utilize existing servers, and 

where appropriate, consolidate servers, data centers and 

common services. 

Department management, however, questioned the 

accuracy of some portions of the rcport. While 

management agreed there will be cost savings from data 

center consolidation opportunities, they felt the savings 

would be significantly less than those stated in the report. 

They believed the calculation used in the report includes 

servers that do not easily lend themselves to virtualization. 

They also believed the calculation included estimated 

savings from early stage virtualization efforts, where 

advantage can be taken of the systems easiest to virtuali~e 

and may not be representative of virtualization savings at 

later stages of the effort. 

Department management also noted that multiple services 

at a certain site can not always be consolidated. They 

stated that providing a few systems with the same services 

may be more appropriate, in some cases, because of user 

needs and efficiency. In addition, rnanagcment believed 

the report advocates consolidating all servers and scientific 

computing into one physical location. Thej stated that this 

scenario is not always cost-effective, there are continuity of 

operation issues involved mi th such consolidations, and 

physical security requirements must be considered. 

------ 

Recommendations and Comments 
AUDITOR NNSA management concurred with our recommendations 

COMMENTS and agreed to take corrective actions that are responsive to 

our recomn~endations. 

While other Department elements concurred nith each of 

the recommendations, action proposed to address 

reconlnlendation one was not fully responsive. In 

particular, management planned to take no additional action 

to share best practices and provide guidance and direction 

regarding data center consolidation. As noted in our report, 

action beyond that currently being taken by program 

officials is necessary to ensure that contractors focus on 

reducing data center costs and energy use. While 

management indicated that it would take action to address 

recommendations two and three, it did not provide specifics 

as to how the corrective actions would be accomplished. 

Based on management's comments on our draft report, we 

asked field sites to correct and provide updated information 

regarding servers. This new data specifically excluded 

research and development and other scientific servers. 

Based on this data, we re-calculated the savings available 

from server virtualization alone and believe that they could 

amount to as much as $2.3 million per year. These 

potential savings are. in our opinion, very conservative in 

that they only include savings available through 

virtualization at five of the Department's sites and do not 

include other savings typically available through 

consolidation. Specifically. cost efficiencies related to data 

center consolidation and reduced lease and maintenance, 

labor, and energy costs were not included in the pro-jected 

savings. 

Both PNNL and Los Alamos reported that they expect to be 

able to eliminate 60 percent of the total servers at their 

respective sites through virtualization. PNNL and other 

laboratories' savings were determined from early stage 

virtualization. For that reason, we used the most 

conservative of all pro-jected savings (reported savings 

ranged from $1,550 to $8,000 per server) in our projected 

savings calculation. 

Contrary to management's conlments, we do not believe 

that one all-purpose system is always the most efficient and 

that there are instances where consolidation may not be 

appropriate. Because of continuity of operations issues, we 

do not advocate consolidating all servers and scientific 

- 

Page 8 Comments 
computing into one physical location. except where 

appropriate. As we noted in the Methodology section of 

this report, we also recognized that there are data centers 

and servers that can not bc consolidated or moved from 

their current location for legitimate business and scientific 

purposes or security considerations. Nonetheless, we 

observed numerous opportunities for consolidating 

duplicative services, such as file serbers, email, data 

storage, and data libraries, services that can. and have been, 

consolidated at some Department sites. 

Page 9 Comments 
Appendix I 

OBJECTIVE To determine whether the Dcpartment of Energy (Department) 

had effectively managed its data center and application hosting 

environments at contractor sties. 

SCOPE 

METHODOLOGY 

The audit was performed between Septembcr 2006 and 

December 2007 at Departmental Headquarters in Washington, 

DC, and Germantown, MD, Argonne National Laboratory in 

Argonne, IL; Fermi National Accelerator Laboratory in 

Batavia, IL; Lawrence Berkeley Laboratory in Berkeley, CA; 

Lawrence Livermore National Laboratory in Livermore, CA; 

Sandia National Laboratory in Albuquerque, IVM; and Los 

Alanlos National Laboratory in Los Alamos, NM. We also 

obtained information from Pacific Northwest National 

Laboratory (PNNL) in Richland, WA. 

For our review. we defined a data center as a room or facility 

used to house computer systems and associated components 

that required special environmental controls, such as extra 

power and cooling. We did not include rooms and servers that 

could not be removed from thcir current location for legitimate 

business and scientific purposes. We also did not include 

supercomputer rooms, as those do not readily lend themselves 

to consolidation. 

To accomplish our objective, we: 

Reviewed applicable laws and directives pertaining to 

information technology management, including the 

Clinger-Cohen Act of 1996 and DOE Order 200.1 ; 

Revicwed applicable standards and requirements issued 

by the National Institute of Standards and Technology; 

Inquired about any data center or application hosting 

guidance and practices throughout the organization; 

Held discussions with field site officials and officials 

from various Dcpartmental offices; and, 

Reviewed reports by the Office of Inspector General 

and the General Accounting Office. 

'To calculate potential savings that could be achieved by 

virtualization of servers. we first examined the savings reported 

-- 

Page 10 Objective, Scope, and Methodology 
Appendix I (continued) 

by four Department entities that had eliminated servers through 

virtualization. The savings identified ranged fi-om 

approximately $1,550 to $8,000 per server. We chose to use as 

our base the most conservative savings, which were reported 

by PNNL. PNNL reported overall savings of $175,000 by 

consolidating 136 servers onto 23 physical hosts, thus saving 

approximately $1,550 per server eliminated. Two field sites 

reported they were able to eliminate, on average. 60 percent of 

the servers through virtualization. Therefore, we calculated our 

savings by first multiplying the 2,433 enterprise servers 

reported by five of the six sites we visited by 60 percent. 

These server counts did not include research and development 

and other scientific computers. The resulting number of 1,460 

servers represents the number of enterprise servers that can 

potentially be eliminated through virtualization at the 5 sites. 

Next, we multiplied the 1,460 servers that could be eliminated 

by $1,550 and determined that the Department could achieve 

savings of approximately $2.3 million at just 5 of the 6 sites we 

reviewed. 

The evaluation was conducted in accordance with generally 

accepted Government auditing standards for performance 

audits and included tests of internal controls and compliance 

with laws and regulations to the extent necessary to satisfy our 

objective. Accordingly, we assessed internal controls 

regarding data centers and application hosting environments. 

Because our review was limited. it would not necessarily have 

disclosed all internal control deficiencies that may have existed 

at the time ofour evaluation. We also evaluated the 

Department's implementation of the Government Perjbrmrrnce 

and Reszalts Act and determined that it had not established 

performance measures for data centers and applications hosting 

environments. We relied on computer processed data and 

validated that data to the extent necessary to satisfy our audit 

objective. 

An exit conference was held with Department officials on 

October 28, 2008. 

- -- 

Page 11 Objective, Scope, and Methodology 
Appendix 2 

PRIOR REI'OR'TS 

Office of Inspector General Reports 

,Ypec*iul Reporl on i24anugemcnt C'h~llleng~s at the Department of'Energy (DOEIIG0782, December 2007). This annual special report detaiIs the Office of Inspector 

General's (OIG) overall concerns. The management challe~lges identified in this report 

include contract management. cyber security, environmental cleanup, human capital 

management. pro-ject management, safeguards and security, and stockpile stewardship. 

In addition, the OIG identified a watch list of areas that are of a concern but do not 

amount to a management challenge. The watch list items are infrastructure 

modernization and worker and comn~unity safety. Furthermore, the OIG identified a 

national trend concerning energy consumption. 

I~ircili~yC'on~ru~~torA~'q~i.sition~~ndM~~n~~g~mentqf'Inji)rrn~~tion TechnologvHurd141rrre 

(DOEIIG-0768, June 2007). Over the past three years, the Department of Energy 

(Department) has spent over $400 million on information technology (IT) hardware; 

however. problems have been noted in its ability to effectively manage its acquisition 

and control of IT hardware. The audit noted that five of seven sites reviewed had not 

developed or fully implemented hardware specifications and brand standards for 

computers and related peripherals, causing at least $4.7 rnillion in unnecessary 

expenditures over the three-year pcriod. The report concluded that the Department 

could potentially realize savings of about $1 6.6 million over five years at the sites 

reviewed by bettcr controlling hardware costs and implementing standards for ccrtain 

equipment 

The Depurtmentl.s Effort., to Implement C'ornmon Infbrmdion Te~.hnology Services at 

Jleudqu~rrter-s (DOEIlG-0763, March 2007). The OIG found that although some 

progress had been made at Headquarters, five major organizations, accounting for 40 

percent of the total potential user population, were not migrated to the common 

operating environment within the first year as planned. thus preventing the realilation 

of the full $1 5 million in expected first year cost savings. The OIG recommended that 

completion of Headquarters' migration to the Department Common Operating 

Environment be accomplished prior to implementation at field sites. Additionally, 

formalization of migration plans, requirements analyses, and cost-benefit analyses 

should be accomplished. 

IMfi,rrntr!ion Technology Supporl Services at lhe Deprrrtm~nt's Operuling Contracto~r. 

(DOEIIG-0725, April 2006). The Department continues to face a number of challenges 

related to contractor procured or furnished IT support services, including contractors 

that did not aggregate demand to leverage or reduce costs; per user costs that varied 

substantially between sites; costs for co-located contractors that varied by as much as 

50 percent per user; and many contractors that did not actively capture or track costs in 

a manner that facilitated Federal oversight. This occurred because the Department had 

Page 12 Prior Reports 
Appendix 2 (continued) 

not established a corporate-wide approach to providing and managing IT support 

services that included contractor-managed sites. In addition, Federal officials charged 

with monitoring contractor performance did not always employ contracting incentives 

to help reduce costs. As a result, there is the potential for significant cost savings at 

contractor-managed sites. For example, we identified the potential for reducing IT 

support costs by ovcr $1 4 million for just the contractor-managed sites co-located at the 

Hanford complex. 

Mun~lgctnenf offhe De~mrfment '.s Desktop C bmputer LC;ojtwure Enterprise License 

Agreernerzls (DOEIIG-07 18, January 2006). This audit was conducted to determine 

whether the Department was effectively managing the acquisition and maintenance of 

desktop software across the agency. Seven of sixteen organizations reviewed were 

found to have acquired software through locally established agreements or contracts at 

prices that were as much as 300 percent higher than those available through the 

Department-level agreements. Although significant savings were pocsible, enterprisewide agreements for conlnlon tools such as security and anti-virus software had not 

been established. Additionally, various locations and organizations acquired 14,000 

encryption software licenses, paid the required annual maintenance fees for a number of 

years: however, never used the licenses. 

Page 13 Prior Reports 
Appendix 3 

Department of Energy 

Natlonal Nuclear Security Adrnlnlrtration 

Washington. DC 20585 

JUL 2 2 2008 

MEMORANDUM FOR Rickey R. Hass 

Assistant Inspector General 

for Environment. Scimce. and Cornorate Audits 

FROM: 

SUBJECT: 

Mictuel c. -ecjC&-- 

Associate Adrninistrato 

for Management and 'Administration 

Comments to Draft Report on Management of Data 

Centers; A06TG038; DRMS No. 2006-29792 

The National Nuclear Security Administration (NNSA) appreciates the 

oppormnity to review the Inspector General's (IG) draft report. "Management of 

Data Caters at Contractor Sites." We understand that, based on the potential to 

reduce costs, the IG initiated this audit to determine whether the 

DepartmmtfNNSA is effectively managing its data centers at its contractor sites. 

We acknowledge the fact that we have not always taken advantage of 

opportunities to improve operational efficiencies. Based on your work. NNSA 

acknowledges the need for the recommendations contained in the draR report. 

Afta considering any unique mission needs and bat business practices, NNSA 

will provide best practice information, guidance, or direction related to 

consolidation; monitor utilization; and, encourage the appropriate consolidation of 

services. 

It is important to note that the reference to Lawrence Livcrmore having programs 

maintain their own centers with little or no coordination of data center services is 

not completely accurate. Ovcr the past year, Lawrence Livermore has built and 

moved into a new institutional business data center and now has the capacity to 

co-locate more of its institutional applications. There arc discussions with 

program entities (National Ignition Facility and the National Atmospheric Release 

Advisoq Cmter, to name two) to explore opportunities to move assets to the new 

data center. Since the data center has only mently opened, not all of the 

ovahead funded servers have been co-located and consolidated. The second half 

of the data center requires increased cooling capacity which is planned for Fiscal 

Year 2009. Lawrence Livumore will consider, at that time, the movement of 

some of the unclassified servas and will consider more consolidation. With any 

consoIidation, prograinmatic need-to-know considerations are driving factors in 

whether scrvas may be consolidated with other servers. Lawrence Livermore is 

Page 14 Management Comments 
Appendix 3 (continued) 

actively pursuing consolidation; therefore the comment that there is little or no 

coordination of data center services between programs is not accurate. 

Should you have any questions about this response, please contact Richard 

Speidel, Director. Policy and Internal Controls Manapment. 

cc: Linda Wilbanks, Chief Information Officer 

Camille Yuan Soo-Hoo, Manager, Livermore Site Ofice 

David Boyd, Senior Procurement Executive 

Karen Boardman, Director, Service Center 

Page 15 Management Comments 
Appendix 3 (continued) 

Department of Energy 

Washington, DC 20585 

August 1,2008 

MEMORANDUM FOR RICKEY R. HASS 

ASSISTANT INSPECTOR GENERAL 

FOR ENVIROMNT, SCIENCE, 

AND CORPORATE AUDITS 

OFFICE OF RJSPECTOR GENERAL . n 

FROM: 

SUBJECT: Response to Inspector General (IG) Draft Report, 

"Management of Data Cente~~ at Contmtor Sites" 

Thank you for the opportunity to review and comment on the draft report, "Management 

of Data Cmtcn at Contractor Sites." The followina is a DeDartmenta) consolidated 

response, which considers comments hm the office of the-chief Information Officer 

(OCIO), the Under Secretary of Energy, and the Office of Science (SC). 

DOE strongly supports the goal of the IG to improve data center efficiency and reduce 

the impact on the environment. However, DOE believes this report misrepresents the 

potential cost savings, the management approaches, and the day to day operations of 

DOE laboratory and site systems. It may not be appropriate to simply apply models from 

industries that are not involved in scientific research and engineering to the DOE 

laboratories and sites. 

The statement that "Without improvements in Department guidance and contractor 

monitoring of server utilization, contractors and field sites will continue to spend more 

than necessary on data center operations." may not be entirely accurate. The laboratories 

and sites are always looking for ways to minimize costs, such as those related to data 

center operations, so that they can maximize the funding available to perform their 

mission work for WE. 

There will be cost savings hm some opportunities for further data center consolidation 

and server virtualization and the labdsites are aggressively pursuing these savings. 

However the cost savings will be significantly less than stated. 

Page 16 Management Comments 
Appendix 3 (continued) 

The projected wst savings in the report is calculated as follows: $7.3 M = 0.6 

(elimination efficiency) X 7,835 (estimated number of servers to be consolidated) 1 1 13 

(number of servers at PNNL used in the example) X $175K (ilaimed savings for PNNL). 

The issue for the above calculation is the 7,835 servers as being indicative for the four 

labs referenced. PNNL would only have 300-400 as their total base number of servers 

that would likely benefit from virtualization. The only way to get to 7,835 servers for 

four labs is to have included scientific servers. Running virtualization software for 

business process servers such as ernail, calendaring, payroll, web servers, etc. is usually 

very effective. Running vittualization software on scientific and engineering applications 

in most cases will actually lower the performance of the server as measured by scientific 

or engineering output. Consequently, the 7,835 estimate is probably wrong by a factor of 

5 to 10 and the real cost savings is a factor of 5 to 10 less than estimated. 

The cost savings estimates bm PNNL and other laboratories were determined from 

relatively early stages virtualization efforts where advantage can be taken of the systems 

easiest to virtualize. It may be more appropriate to ust a methodology that recognizes 

this factor with a virtualization maturity model. 

At multiple times, the report implies that because there are multiple instances of certain 

services at a single laboratory, then are necessarily consolidation opportunities. This is 

not necessarily always true. The laboratories and sites, with the exceptions of some 

limited administrative systems, are not comparable to large corporations or government 

agencies. It is not always the case that providing one all-purpose system is actually more 

efficient than providing a few limited purpose systcms because the need to provide all 

possible features and platforms actually trades off with efficiency. 

This report may be misleading in how it defines a "data center." It appears to be claiming 

tbat consolidating all servers and scientific computing in one physical place is clearly the 

most cost effective. The laboratories and sites attempt to consolidate data centers 

whenever possible. In many cases the up-front costs of space, fiber network distribution, 

power distribution and cooling systems are tbe driving factors rather than the costs of 

computing and networking systems thcmsclvcs. To be cost effective, this situation must 

be analyzed location by location so that advantages can be taken for site specific 

resources and mission requirements. Most laboratories and sites run several physical 

locations as a vid data center or centers as this sometimes provides the most cost 

effective operational solution when all costs are included. In addition, continuity of 

options considerations usually require at least two or more physical locations be used. 

The physical security requirements for providing appropriate cyber security must also be 

taken into consideration. Specifically, the physical locations and separations required to 

meet certain cyber security requirements must be taken into account in any concept of 

data center consolidation. 

Page 17 Management Comments 
Appendix 3 (continued) 

Recommendation 1: To help improve the efficiency of contractor-operated data centers, 

share best practices and provide guidance and direction regarding data center 

consolidation through application hosting and server consolidation. 

Management Response: 

DOE concurs with sharing best practices and providing guidance and direction regarding 

data center consolidation through application hosting and server consolidation. 

On sharing best practices, the DOE National Laboratories CIOs (NLCIO) meet three or 

four times a year and Energy, National Nuclear Security Administration, SC, and DOE 

OCIO staffs are frequently present. Roughly half of the meeting time is devoted to 

sharing IT best practices. Data center consolidation and server virtualization projects 

have been subjects of this lT best practices sharing multiple times over the past several 

years. 

DOE already performs numerous reviews on achieving mission program goals and 

laboratory and site operating efficiencies which includes the IT programs and activities at 

the laboratories and sites. As warranted by the results of these reviews in the IT area, 

DOE will provide appropriate guidance and direction regarding data center consolidation 

through application hosting and server consolidation. 

The existing NLCIO, WE program offices and OCIO meetings are already providing a 

forum for sharing best practices for data center consolidation and deployment of server 

virtualization technologies. 

Estimated Completion Date: 

Completed. No new action is required at this time, but guidance and direction will be 

issued as needed per the discussion above. 

Recommendation 2: Require contractors and field sites to actively monitor server 

utilization and take action to more hlly utilize existing server capacity. 

Management Response: 

DOE concurs with this recommendation and will verify that the laboratories and sites are 

appropriately consolidating data centers and deploying server virtualization technologies 

including its utilization that optimize the cost eEciencies in this area for achieving their 

DOE mission. 

Page 18 Management Comments 
Appendix 3 (continued) 

Action Plan: 

DOE will have all of its laboratories and sites validate they are optimizing the 

consolidation of their data centers and deployment of smer virtualhation technologies 

including utilization monitoring. 

Estimated Completion Date: 

All DOE laboratoricr and sites will pvi& the validation by no later than 

Mmh 3 1,2009, to thcu responsible DOE ficld/sitc office. 

Rtcommendation 3: Require contractors and field sites to consolidate savers, data 

centers, and common services, whm appropriaic. 

Management Response: 

DOE concurs with this recommendation and will verify that the laboratories and sites arc 

appropriately consolidating data centers and deploy&server virtualization technologies 

including its utilization that optimize the cost efficiencies in this area for achieving their 

DOE mission. 

Action Plan: 

WE will have all of its Laboratories and sites validak they are optimizing the 

consolidstion of their data centers and deployment of scrver virtuollization technologics 

including utilization monitoring. 

-ted Completion D.b: All DOE laboratories and sites will provide the validation 

by no later than March 3 1,2009, to their mponsible DOE field/sitc office. 

If you have any questions related to this response, please contact Devon Streit at 202- 

586-9129. 

Page 19 Management Comments 
IG Report No. DOEIIG-0803 

CUSTOMER RESPONSE FORM 

The Office of lnspector Gencral has a continuing interest in improving the usefulness of 

its products. We wish to make our reports as responsive as possible to our customers' 

requirements, and, therefore, ask that you consider sharing your thoughts with us. On the 

back of this form. you may suggest improvements to enhance thc effectiveness of future 

reports. Please include answers to the following questions if they are applicable to you: 

1. What additional background information about the selection, scheduling, scope, or 

procedures of the inspection would have been helpful to the reader in understanding 

this report? 

2. What additional information related to findings and recommendations could have 

been included in the report to assist management in implementing corrective actions? 

3. What format, stylistic, or organizational changes might have made this report's 

overall message more clear to the reader? 

4. What additional actions could the Office of lnspector General have taken on the 

issues discussed in this report which would have been helpful? 

5. Please include your name and telephone number so that we may contact you should 

we have any questions about your comments. 

Name Date 

Telephone Organization 

When you have completed this form. you may telefax it to the Office of lnspector 

General at (202) 586-0948, or you may mail it to: 

Office of Inspector General (IG-I) 

Department of Energy 

Washington, DC 20585 

ATTN: Customer Relations 

If you wish to discuss this report or your comments with a staff member of the Office of 

lnspector General, please contact Judy Garland-Smith (202) 586-7828. 
The Office of Inspector General wants to make the distribution of its reports as customer friendly 

and cost effective as possible. Therefore, this report will be available electronically through the 

Internet at the following address: 

I1.S. Department of Energy Office of Inspector General Home Page 

htlp://~ww.ig.energy.gov 

Your comments would be appreciated and can be provided on the Customer Response Form.