NREL is a national laboratory of the U.S. Department of Energy, Office of Energy Efficiency and Renewable Energy, operated by the Alliance for Sustainable Energy, LLC.

Trends in Data Center Design – 

ASHRAE Leads the Way to 

Large Energy Savings

ASHRAE Conference, Denver

Otto Van Geet, PE

June 24, 2014

NREL/PR-6A40-58902
2

Data Center Energy

Data centers are energy intensive facilities.

• 10-100x more energy intensive than an office 

• Server racks well in excess of 30 kW

• Surging demand for data storage 

• EPA estimate: 3% of U.S. electricity

• Power and cooling constraints in existing 

facilities.

Data center inefficiency steals power that 

would otherwise support compute capability.

Photo by Steve Hammond, NREL
3

BPG Table of Contents • Summary • Background • Information Technology Systems • Environmental Conditions • Air Management • Cooling Systems • Electrical Systems • Other Opportunities for 

Energy Efficient Design

• Data Center Metrics & Benchmarking
4

CPUs

~65C

(149F)

GPUs

~75C

(167F) Memory 

~85C

(185F)

CPU, GPU & Memory, represent ~75-90% of heat load ...





Safe Temperature Limits

Photo by Steve Hammond, NREL
5

Data center equipment’s environmental conditions should fall 

within the ranges established by ASHRAE as published in the 

Thermal Guidelines. 

Environmental Conditions

ASHRAE Reference: ASHRAE (2008), (2011) 

(@ Equipment Intake) Recommended Allowable

Temperature

 Data Centers ASHRAE 65°

 – 80°F 59°

 – 90°F (A1)

41° – 113°F (A4)

Humidity (RH)

 Data Centers ASHRAE 

42°F DP – 

60% or 59oF DP

20% – 

80% & 63°F DP

Environmental Specifications (°F)
6

Equipment Environmental Specification 

Air inlet to IT equipment is the 

important specification to meet

Outlet temperature is not 

important to IT equipment
7

Recommended Range (Statement of Reliability)

Preferred facility operation; most values should be within this range.

Allowable Range (Statement of Functionality)

Robustness of equipment; no values should be outside this range.

MAX ALLOWABLE

RACK INTAKE

TEMPERATURE

MAX RECOMMENDED

Over-Temp

Recommended 

Range

Under-Temp

MIN RECOMMENDED

MIN ALLOWABLE

Allowable 

Range

Key Nomenclature
8

2011 ASHRAE Thermal Guidelines

2011 Thermal Guidelines for Data Processing Environments – Expanded Data Center Classes and 

Usage Guidance. White paper prepared by ASHRAE Technical Committee TC 9.9
9

2011 ASHRAE Allowable Ranges

Dry Bulb Temperature
10

Psychrometric Bin Analysis

0

0.002

0.004

0.006

0.008

0.01

0.012

0.014

0.016

0.018

0.02

0.022

0.024

0.026

0.028

0.03

0.032

0.034

0.036

30 40 50 60 70 80 90 100 110 120

Humidity Ratio (lb Water/lb Dry Air)

Dry-Bulb Temperature (oF)

Boulder, Colorado TMY3 Weather Data

TMY3 Weather Data

Class 1 Recommended Range

Class 1 Allowable Range

60oF 

50oF 

40oF 

Relative Humidity 80%

60%

40%

20%

100%

70oF 

60oF 

50oF 

40oF 

Relative Humidity 80%

60%

40%

20%

100%

80oF 

Design Conditions (0.4%):

91.2 db, 60.6 wb
11

Estimated Savings

Baseline

System DX Cooling with no economizer

Load 1 ton of cooling, constant year-round

Efficiency (COP) 3

Total Energy (kWh/yr) 10,270

RECOMMENDED RANGE ALLOWABLE RANGE

Results Hours Energy (kWh) Hours Energy (kWh)

Zone1: DX Cooling Only 25 8 2 1

Zone2: Multistage Indirect Evap. + DX (H80) 26 16 4 3

Zone3: Multistage Indirect Evap. Only 3 1 0 0

Zone4: Evap. Cooler Only 867 97 510 57

Zone5: Evap. Cooler + Economizer 6055 417 1656 99

Zone6: Economizer Only 994 0 4079 0

Zone7: 100% Outside Air 790 0 2509 0

Total 8,760 538 8,760 160

Estimated % Savings - 95% - 98%
12

Energy Savings Potential: Economizer Cooling

Energy savings potential for recommended 

envelope, Stage 1: Economizer Cooling.12

(Source: Billy Roberts, NREL) 
13

Energy Savings Potential: 

Economizer + Direct Evaporative Cooling

Energy savings potential for recommended envelope, 

Stage 2: Economizer + Direct Evap. Cooling.12 

(Source: Billy Roberts, NREL) 
14

Energy Savings Potential: Economizer + Direct 

Evap. + Multistage Indirect Evap. Cooling

Energy savings potential for recommended 

envelope, Stage 3: Economizer + Direct Evap. 

+ Multistage Indirect Evap. Cooling.12 

(Source: Billy Roberts, NREL) 
15

Improve Air Management

• Typically, more air 

circulated than required 

• Air mixing and short 

circuiting leads to:

– Low supply temperature

– Low Delta T

• Use hot and cold aisles

• Improve isolation of hot 

and cold aisles

– Reduce fan energy

– Improve air-conditioning 

efficiency

– Increase cooling capacity

15

Hot aisle/cold aisle configuration 

decreases mixing of intake & 

exhaust air, promoting efficiency. 

Source: http://www1.eere.energy.gov/femp/pdfs/eedatacenterbestpractices.pdf
16

Isolate Cold and Hot Aisles

Source: http://www1.eere.energy.gov/femp/pdfs/eedatacenterbestpractices.pdf

70-80oF vs. 45-55oF

95-105oF vs. 60-70oF
17

Adding Air Curtains for Hot/Cold Isolation

Photo used with permission from the National Snow and Ice Data Center. http://www.nrel.gov/docs/fy12osti/53939.pdf
18

Move to Liquid Cooling

• Server fans are inefficient and noisy.

– Liquid doors are an improvement but we can do 

better!

• Power densities are rising making componentlevel liquid cooling solutions more appropriate.

• Liquid benefits

– Thermal stability, reduced component failures.

– Better waste heat re-use options. 

– Warm water cooling, reduce/eliminate 

condensation.

– Provide cooling with higher temperature coolant.

• Eliminate expensive & inefficient chillers.

• Save wasted fan energy and use it for 

computing.

• Unlock your cores and overclock to increase 

throughput!
19

Liquid Cooling – Overview

Water and other liquids (dielectrics, glycols and refrigerants) 

may be used for heat removal. 

• Liquids typically use LESS transport energy 

(14.36 Air to Water Horsepower ratio for example below). 

• Liquid-to-liquid heat exchangers have closer approach temps than 

Liquid-to-air (coils), yielding increased economizer hours.
20

Liquid Cooling – New Considerations

• Air Cooling

– Humidity 

– Fan failures

– Air side economizers, particulates

• Liquid Cooling

– pH & bacteria

– Dissolved solids

– Corrosion inhibitors, etc.

• When considering liquid-cooled 

systems, insist that providers adhere 

to the latest ASHRAE water quality 

spec or it could be costly.
21

2011 ASHRAE Liquid Cooling Guidelines
22

2011 ASHRAE Liquid Cooling Guidelines

NREL ESIF HPC (HP hardware) using 75 F supply, 113 F return –W4/W5
Courtesy of Henry Coles, Lawrence Berkeley National Laboratory
24

Three Cooling Device Categories

IT

Equipment

Rack

cooling 

water

rack

containment

SERVER

FRONT

1 - Rack Cooler

• APC-water

• Knürr(CoolTherm)-water

• Knürr(CoolLoop)-water

• Rittal-water

2 - Row Cooler

• APC(2*)-water

• Liebert-refrigerant

IT

Equipment

Rack

IT

Equipment

Rack

IT

Equipment

Rack

row

containment

cooling 

water

cooling 

water SERVER

FRONT

3 - Passive Door Cooler

• IBM-water

• Vette/Coolcentric-water

• Liebert-refrigerant

• SUN-refrigerant

SERVER

FRONT

IT

Equipment

Rack

cooling 

water

Courtesy of Henry Coles, Lawrence Berkeley National Laboratory
25

“Chill-off 2” Evaluation of Close-coupled 

Cooling Solutions 

Courtesy of Geoffrey Bell and Henry Coles, Lawrence Berkeley National Laboratory

less energy

use
26

Cooling Takeaways...

• Use a central plant (e.g., chiller/CRAHs vs. CRAC units)

• Use centralized controls on CRAC/CRAH units to prevent 

simultaneous humidifying and dehumidifying

• Move to liquid cooling (room, row, rack, chip)

• Consider VSDs on fans, pumps, chillers, and towers

• Use air- or water-side economizers

• Expand humidity range and improve humidity control (or 

disconnect)
27

Data Center Efficiency Metric

• Power Usage Effectiveness (PUE) is an industry standard data 

center efficiency metric.

• The ratio of power used or lost by data center facility 

infrastructure (pumps, lights, fans, conversions, UPS...) to 

power used by compute.

• Not perfect, some folks play games with it.

• 2011 survey estimates industry average is 1.8.

• Typical data center, half of power goes to things other than 

compute capability.

27

“IT power” + “Facility power” 

P.U.E. = “IT power” 
28

PUE – Simple and Effective
29

-20 0

20

40

60

80

100

0.75

0.85

0.95

1.05

1.15

1.25

1.35

1.45

Outdoor Temperature (°F)

PUE

Data Center PUE

-20 0

20

40

60

80

100

0.75

0.85

0.95

1.05

1.15

1.25

1.35

1.45

Outdoor Temperature (°F)

PUE

Data Center PUE

Outdoor Temperature

NREL PIX 17828
“I am re-using waste 

heat from my data 

center on another 

part of my site and 

my PUE is 0.8!”

http://www.thegreengrid.org/en/Global/Content/white-papers/ERE
“I am re-using waste 

heat from my data 

center on another 

part of my site and 

my PUE is 0.8!”

http://www.thegreengrid.org/en/Global/Content/white-papers/ERE
http://www.thegreengrid.org/en/Global/Content/white-papers/ERE

ASHRAE & friends (DOE, EPA, 

TGG, 7x24, etc..) do not allow 

reused energy in PUE & PUE is 

always >1.0.

Another metric has been 

developed by The Green Grid +; 

ERE – Energy Reuse Effectiveness.
33

ERE – Adds Energy Reuse

Utility

Cooling

UPS PDU

IT

Rejected

Energy

(a)

(b)

(c) (d)

(f)

(e)

Reused

Energy

(g)
34

Credit: Haselden Construction

• More than 1300 people in DOE 

office space on NREL’s campus

• 360,0,000 ft2 

• Design/build process with 

required energy goals ̶ 25 kBtu/ft2

̶ 50% energy savings

̶ LEED Platinum

• Replicable ̶ Process 

̶ Technologies

̶ Cost

• Site, source, carbon, cost ZEB:B ̶ Includes plugs loads and datacenter

• Firm fixed price - $246/ft2

construction cost (not including 

$27/ft2 for PV from PPA/ARRA)

• Opened June 10, 2010 (First 

Phase)

DOE/NREL Research Support Facility 
35

RSF Datacenter

• Fully containing hot aisle

– Custom aisle floor and door seals

– Ensure equipment designed for 

cold

aisle containment 

 And installed to pull cold air

 Not hot air...

– 1.18 annual PUE 

– ERE = 0.9 

• Control hot aisle based on

return temperature of ~90F.

• Waste heat used to heat building.

• Economizer and Evaporative 

Cooling 

• Low fan energy design

• 1900 ft2 

Credit: Marjorie Schott/NREL

NREL PIX 25417
36

NREL HPC Data Center

Showcase Facility

• 10MW, 10,000 ft2 

• Leverage favorable climate

• Use evaporative rather 

mechanical cooling. 

• Waste heat captured and used 

to heat labs & offices.

• World’s most energy efficient 

data center, PUE 1.06!

• Lower CapEx and OpEx. 

Leveraged expertise in energy efficient 

buildings to focus on showcase data center

Chips to bricks approach

• Petascale+ HPC Capability in 2013

• 20-year planning horizon

̶ 5 to 6 HPC generations.

̶ Insight Center

 Scientific data visualization

 Collaboration and interaction. 

High 

Performance 

Computing

 
37

Critical Data Center Specs

• Warm water cooling, 75F (24C) ̶ Water much better working fluid than air 

- pumps trump fans.

̶ Utilize high quality waste heat, 95F (35C) 

or warmer.

̶ +90% IT heat load to liquid.

• High power distribution

̶ 480VAC, Eliminate conversions.

• Think outside the box

̶ Don’t be satisfied with an energy efficient 

data center nestled on campus 

surrounded by inefficient laboratory and 

office buildings.

̶ Innovate, integrate, optimize.

Dashboards report instantaneous, seasonal, and cumulative PUE values. 

Photo by Steve Hammond, NREL
38

• Thermoelectric power generation (coal, oil, 

natural gas and nuclear) consumes about 1.1 

gallon per kW hour, on average.

• This amounts to about 9.5M gallons per MW 

year.

• We estimate about 2.0M gallons water 

consumed per MW year for on-site evaporative 

cooling at NREL.

• If chillers need 0.2MW per MW of HPC power, 

then chillers have an impact of 2.375M gallons 

per year per MW.

• Actuals will depend on your site, but evap. 

cooling doesn’t necessarily result in a net 

increase in water use. 

Water Considerations

“We shouldn’t use evaporative cooling, water is scarce.”

NREL PIX 17828
39

Data Center Energy Efficiency

• ASHRAE 90.1 2011 requires economizer in most data centers.

• ASHRAE Standard 90.4P, Energy Standard for Data Centers and 

Telecommunications Buildings 

• PURPOSE: To establish the minimum energy efficiency requirements 

of data centers and telecommunications buildings for: 

– Design, construction, and a plan for operation and maintenance 

• SCOPE: This standard applies to: 

– New, new additions, and modifications to data centers and telecommunications 

buildings or portions thereof and their systems 

• Will set minimum PUE based on climate

• More details at https://www.ashrae.org/news/2013/ashrae-seeksinput-on-revisions-to-data-centers-in-90-1-energy-standard-scope
40

1. Reduce the IT load - Virtualization & Consolidation (up to 80% reduction). 

2. Implement contained hot aisle and cold aisle layout.

̶ Curtains, equipment configuration, blank panels, cable entrance/exit ports, 

3. Install economizer (air or water) and evaporative cooling (direct or indirect).

4. Raise discharge air temperature. Install VFD’s on all computer room air 

conditioning (CRAC) fans (if used) and network the controls.

5. Reuse data center waste heat if possible.

6. Raise the chilled water (if used) set-point.

̶ Increasing chiller water temperature by 1°F reduces chiller energy use by 1.4% 

7. Install high efficiency equipment including UPS, power supplies, etc.

8. Move chilled water as close to server as possible (direct liquid cooling).

9. Consider centralized high efficiency water cooled chiller plant

̶ Air-cooled = 2.9 COP, water-cooled = 7.8 COP.

Energy Conservation Measures
41

Otto VanGeet

303.384.7369

Otto.VanGeet@nrel.gov

 

RSF II 

21 kBtu/ft2 

$246/ft2 Construction Cost 

QUESTIONS?