Lessons Learned from

the World’s Most EnergyEfficient Data Center

Otto Van Geet, PE

Principal Engineer

EUCI Utilities, Data Centers, and Renewable 

Energy Summit

July 25, 2018

NREL/PR-7A40-71928
NREL | 2

NREL’s Dual Computing Mission

• Provide HPC and related systems expertise to advance NREL’s mission, 

and push the leading edge for data center sustainability

• Demonstrate leadership in liquid cooling, waste heat capture, and re-use

• Holistic “chips to bricks” approaches to data center efficiency

• Showcase data center at NREL’s Energy Systems Integration Facility (ESIF)

Critical Topics Include: 

• Liquid cooling and energy efficiency

• Water efficiency
NREL | 3

Planning for a New Data Center

• Started planning for new data center in 2006

• Based on HPC industry/technology trends, committed to direct liquid cooling

• Holistic approach: integrate racks into the data center, data center into the facility,

the facility into the NREL campus

• Capture and use data center waste heat: office and lab space (now) and export to

campus (future)

• Incorporate high power density racks—60kW+ per rack

• Implement liquid cooling at the rack, no mechanical chillers

• Use chilled beam for office/lab space heating. Low-grade waste heat use

• Considered two critical temperatures:

– Information technology (IT) cooling supply—could produce 24oC (75oF) on

hottest day of the year, ASHRAE “W2” class

– IT return water—required 35oC (95oF) to heat the facility on the coldest day

of the year

Build the World’s Most Energy Efficient Data Center
NREL | 4

NREL Data Center

Showcase Facility

• ESIF 182,000 ft.2 research facility

• 10,000 ft.2 data center

• 10-MW at full buildout

• LEED Platinum Facility, PUE ≤ 1.06

• NO mechanical cooling (eliminates 

expensive and inefficient chillers)

Data Center Features

• Direct, component-level liquid cooling, 

24oC (75oF) cooling water supply

• 35-40oC (95-104oF) return water (waste 

heat), captured and used to heat offices 

and lab space

• Pumps more efficient than fans

• High voltage 480-VAC power distribution 

directly to high power density 60-80-kW 

compute racks

Compared to a Typical Data Center

• Lower CapEx—costs less to build

• Lower OpEx—efficiencies save

Utilize the bytes and the BTUs! Integrated “Chips to Bricks” 

Approach
NREL | 5

Cooling Efficiency

• Liquid conduits require 250 to 1,000 times less space than air conduits for 

transporting the same quantity of heat energy.

• Liquids require 10 to 20 times less energy to transport energy.

• Liquid-to-liquid heat exchangers have closer approach temperatures than liquidto-air (coils), yielding greater efficiency and increased economizer hours.

• ASHRAE TC9.9 liquid standards provide excellent guide.
NREL | 6

Energy Efficient Data Centers

• Choices regarding power, packaging, 

cooling, and energy recovery in data 

centers drive total cost of ownership.

• Why should we care?

o Water usage

o Limited utility power

o Mega$ per MW year

o Cost: OpEx ~ IT CapEx!

• Space Premium: Ten 100-kW racks take 

much less space than the equivalent 50 

20-kW air cooled racks.
NREL | 7

Metrics

PUE = “Facility energy” + “IT energy”

“IT energy”

ERE = “Facility energy” + “IT energy” – “Reuse energy”

“IT energy”
NREL | 8

Metrics

WUE = “Annual Site Water Usage”

“IT energy”

the units of WUE are liters/kWh

WUESOURCE =

“Annual Site Water Usage” + “Annual Source Energy Water Usage”

“IT energy”

WUESOURCE = 

“Annual Site Water Usage”

“IT energy” + [EWIF × PUE] 

where EWIF is energy water intensity factor
NREL | 9

System Schematic: Original Configuration

4th Floor

Mechanical

2nd Floor

Mechanical

3rd Floor

Data Center
NREL | 10

Air-Cooled to Liquid-Cooled Racks

Traditional air-cooled 

allow for rack power 

densities of 1kW-5kW

Require liquid-cooled when rack 

power densities in 5kW to 80kW 

range, have several options
NREL | 11

Liquid-Cooled Server Options

Cold Plates

Peregrine HPC Tray
NREL | 12

Fanless Liquid-Cooled Server Options

Direct Immersion

Fixed Cold Plate
NREL | 13

Data Center Water Distribution

To/From 4th Floor

Mechanical Room
NREL | 14

Liquid Cooling—Considerations

• Liquid cooling essential at high power density

• Compatible metals and water chemistry is crucial

• Cooling distribution units (CDUs)

o Efficient heat exchangers to separate facility and server liquids

o Flow control to manage heat return

o System filtration (with bypass) to ensure quality

• Redundancy in hydronic system (pumps, heat exchangers)

• Plan for hierarchy of systems

o Cooling in series rather than parallel

o Most sensitive systems get coolest liquid

• At least 95% of rack heat load captured directly to liquid
NREL | 15

Air- and Water-Cooled System Options

Air-Cooled System

• Design day is based on DRY BULB 

temperature

• Consumes no water

(no evaporative cooling)

• Large footprint/requires very large 

airflow rates

Water-Cooled System

• Design day is based on the lower WET BULB 

temperature

• Evaporative cooling process uses water to improve 

cooling efficiency

o 80% LESS AIRFLOW  lower fan energy

o Lower cost and smaller footprint.

• Colder heat rejection temperatures improve 

system efficiency

However, water-cooled 

systems depend on a 

reliable, continuous 

source of low-cost water.
NREL | 16

Traditional Wet Cooling System

95°F (35.0°C)

Process Loop

Heat In

Moist Heat 

Out

Wet Loop

Sized for Design Day

Thermal Duty

75°F (23.9°C)

95°F (35.0°C) 75°F (23.9°C)

Condenser Water Pump
NREL | 17

Basic Hybrid System Concept

95°F (35.0°C)

Process Loop

Heat In

Moist Heat 

Out

Wet Loop

Sized for Design Day

Thermal Duty

75°F (23.9°C)

75°F (23.9°C)

Condenser Water Pump

Process Loop

Dry Loop

Sized for Water Savings

85°F (29.4°C)

Dry Heat 

Out

“Wet” when it’s hot, “dry” when it’s not.

Dry Sensible Cooler

95°F (35.0°C)
NREL | 18

Thermosyphon Cooler

Process 

Water In

Out to 

Tower
NREL | 19

Improved WUE—Thermosyphon
NREL | 20

System Modeling Program
NREL | 21

Applications

Favorable Application Characteristics

• Year-round heat rejection load (24/7, 365 days is best)

• Higher loop temperatures relative to average ambient temperatures

• High water and wastewater rates or actual water restrictions

• Owner’s desire to mitigate risk of future lack of continuous water availability 

(water resiliency)

• Owner’s desire to reduce water footprint to meet water conservation targets

Any application using an open cooling tower is a 

potential application for a hybrid cooling system, but 

certain characteristics will increase the potential for 

success.
NREL | 22

Sample Data: Typical Loads and Heat Sinks
NREL | 23

Cumulative Water and Cost Savings

Energy = $0.07/kWh

Water = $5.18/kgal
NREL | 24

Data Center Metrics

PUE = 1.034 

ERE = 0.929 

WUE = 0.7 liters/kWh 

First year of TSC operation (9/1/2016–8/31/2017) 

Hourly average IT Load

= 888 kW 

WUESOURCE = 5.4 liters/kWh

WUESOURCE = 4.9 liters/kWh if energy from 

720 kW PV (10.5%) is included

using EWIF 4.542 liters/kWh for Colorado

Annual Heat Rejection

(with only cooling towers, WUE = 1.42 liters/kWh) 
NREL | 25

Conclusions

• Warm-water liquid cooling has proven very energy

efficient in operation

• Modeling of a hybrid system showed that it was possible

to save significant amounts of water while simultaneously

reducing total operating costs

o System modification was straightforward

o System water and operational cost savings are in line

with modeling

• Hybrid system increased operational resiliency
NREL | 26

Questions

720 KW

1,156 KW

524 KW

449 KW

408 KW 94 KW

NREL PV Systems -

South Table Mesa 

Campus

50 KW

Otto VanGeet, 303-384-7369, otto.vangeet@nrel.gov
NREL | 27

Notice

This research was performed using computational resources sponsored by the U.S. 

Department of Energy's (DOE) Office of Energy Efficiency and Renewable Energy 

and located at the National Renewable Energy Laboratory under Contract No. DEAC36-08GO28308. Funding provided by the Federal Energy Management 

Program. The views expressed in the presentation do not necessarily represent 

the views of the DOE or the U.S. Government. The U.S. Government retains and 

the publisher, by accepting the presentation for publication, acknowledges that 

the U.S. Government retains a nonexclusive, paid-up, irrevocable, worldwide 

license to publish or reproduce the published form of this work, or allow others to 

do so, for U.S. Government purposes.