AUGUST 21 - 23, 2018 • CLEVELAND, OHIO
Liquid in the Rack: Liquid Cooling 

Your Data Center

NREL ESIF Data Center

Otto Van Geet, PE – NREL

NREL/PR-7A40-72046
NREL | 3

NREL’s Dual Computing Mission

• Provide HPC and related systems expertise to advance NREL’s mission, 

and push the leading edge for data center sustainability

• Demonstrate leadership in liquid cooling, waste heat capture, and re-use

• Holistic “chips-to-bricks” approaches to data center efficiency

• Showcase data center at NREL’s Energy Systems Integration Facility (ESIF)

Critical Topics Include: 

• Liquid cooling and energy efficiency

• Water efficiency
NREL | 4

Planning for a New Data Center

• Started planning for new data center in 2006

• Based on HPC industry/technology trends, committed to direct liquid cooling

• Holistic approach: integrate racks into the data center, data center into the facility, 

the facility into the NREL campus

• Capture and use data center waste heat: office and lab space (now) and export to 

campus (future)

• Incorporate high-power density racks—more than 60 kW per rack

• Implement liquid cooling at the rack, no mechanical chillers

• Use chilled beam for office/lab space heating. Low-grade waste heat use.

• Considered two critical temperatures:

– Information technology (IT) cooling supply—could produce 24oC (75oF) on 

hottest day of the year, ASHRAE “W2” class

– IT return water—required 35oC (95oF) to heat the facility on the coldest day 

of the year

Build the World’s Most Energy Efficient Data Center
NREL | 5

NREL Data Center

Showcase Facility

• ESIF 182,000 ft.2 research facility

• 10,000 ft.2 data center

• 10-MW at full buildout

• LEED Platinum Facility, PUE ≤ 1.06

• NO mechanical cooling (eliminates 

expensive and inefficient chillers)

Data Center Features

• Direct, component-level liquid cooling, 

24oC (75oF) cooling water supply

• 35-40oC (95-104oF) return water (waste 

heat) is captured and used to heat offices 

and lab space

• Pumps more efficient than fans

• High-voltage, 480-VAC power distribution 

directly to high power density 60- to 80-

kW compute racks

Compared to a Typical Data Center

• Lower CapEx—costs less to build

• Lower OpEx—efficiencies save

Utilize the bytes and the BTUs! Integrated “Chips-to-Bricks” 

Approach
NREL | 6

Metrics

PUE = “Facility energy” + “IT energy”

“IT energy”

ERE = “Facility energy” + “IT energy” – “Reuse energy”

“IT energy”
NREL | 7

Metrics

WUE = “Annual Site Water Usage”

“IT energy”

the units of WUE are liters/kWh

WUESOURCE =

“Annual Site Water Usage” + “Annual Source Energy Water Usage”

“IT energy”

WUESOURCE = 

“Annual Site Water Usage”

“IT energy” + [EWIF × PUE] 

where EWIF is energy water intensity factor
NREL | 8

Air-Cooled to Liquid-Cooled Racks

Traditional air-cooled 

allow for rack power 

densities of 1kW-5kW

Require liquid-cooled when rack 

power densities in 5–80kW range, 

have several options
NREL | 9

Liquid-Cooled Server Options

Cold Plates

Peregrine HPC Tray
NREL | 10

Fanless Liquid-Cooled Server Options

Direct Immersion

Fixed Cold Plate
NREL | 11

Data Center Water Distribution

To/From 4th Floor

Mechanical Room
NREL | 12

Liquid Cooling—Considerations

• Liquid cooling essential at high-power density

• Compatible metals and water chemistry is crucial

• Cooling distribution units (CDUs)

o Efficient heat exchangers to separate facility and server liquids

o Flow control to manage heat return

o System filtration (with bypass) to ensure quality

• Redundancy in hydronic system (pumps, heat exchangers)

• Plan for hierarchy of systems

o Cooling in series rather than parallel

o Most sensitive systems get coolest liquid

• At least 95% of rack heat load captured directly to liquid
NREL | 13

Basic Hybrid System Concept

95°F (35.0°C)

Process Loop

Heat In

Moist Heat 

Out

Wet Loop

Sized for Design Day

Thermal Duty

75°F (23.9°C)

75°F (23.9°C)

Condenser Water Pump

Process Loop

Dry Loop

Sized for Water Savings

85°F (29.4°C)

Dry Heat 

Out

“Wet” when it’s hot, “dry” when it’s not.

Dry Sensible Cooler

95°F (35.0°C)

Heat 

Exchanger

Tower
NREL | 14

Improved WUE—Thermosyphon
NREL | 15

ESIF Data Center Efficiency Dashboard 
NREL | 16

Data Center Metrics

PUE = 1.034 

ERE = 0.929 

WUE = 0.7 liters/kWh 

First year of TSC operation (9/1/2016–8/31/2017) 

Hourly average IT Load

= 888 kW 

WUESOURCE = 5.4 liters/kWh

WUESOURCE = 4.9 liters/kWh if energy from 

720 kW PV (10.5%) is included

using EWIF 4.542 liters/kWh for Colorado

Annual Heat Rejection

(with only cooling towers, WUE = 1.42 liters/kWh) 
Otto Van Geet, PE

Principal Engineer, NREL

Otto.vangeet@nrel.gov
NREL | 18

Notice

This research was performed using computational resources sponsored by the 

U.S. Department of Energy's (DOE) Office of Energy Efficiency and Renewable 

Energy and located at the National Renewable Energy Laboratory under Contract 

No. DE-AC36-08GO28308. Funding provided by the Federal Energy Management 

Program. The views expressed in the presentation do not necessarily represent 

the views of the DOE or the U.S. Government. The U.S. Government retains and 

the publisher, by accepting the presentation for publication, acknowledges that 

the U.S. Government retains a nonexclusive, paid-up, irrevocable, worldwide 

license to publish or reproduce the published form of this work, or allow others 

to do so, for U.S. Government purposes.