Resource Management: Assessing theWarm Water Facilities and Systems
Supporting ORNL’s Summit
Jim Rogers
Director, Computing and Facilities
National Center for Computational Sciences
Oak Ridge National Laboratory

ORNL is managed by UT-Battelle, LLC for the US Department of Energy

ORNL Leadership Class Systems 2004 - 2018

27
PF

18.5
TF

25
TF

54
TF

2006
Cray XT32004
2005
Cray X1E Cray XT3 Jaguar
Phoenix Jaguar
262
TF

263
TF

2008
Cray XT4
2007 Jaguar
Cray XT4Jaguar

2.5
PF1PF

2008
Cray XT5Jaguar

2012
Cray XK7Titan

2009
Cray XT5Jaguar

From 2004 – 2018, HPC systems relied onchiller-based cooling (5.5°C / 42°Fsupply) with annualized PUEs to ~1.4

ORNL’s Transition to Warmer Facility Supply Temperatures
Titan: Refrigerant-based
per-rack cooling withdirect rejection of heat tocold 5.5°C water• Below dewpoint• 100% use of chillers

Summit: A combination of
direct on-package coolingand RDHX with 21°C / 70°F
supply is > 95% roomneutral.
• Above dewpoint• Contribution by chillers~20% of the hours of theyear

Frontier: Custom packaging is>95% room-neutral with a 32°C /90°F supply.
• ~100% Evaporative Cooling,with supplemental HVAC forparasitic loads

~1.5
EF200
PF
27
PF32012
Cray XK7Titan

2018/2019
IBMSummit

2021/2022
HPE/CrayFrontier

Motivation for “Warmer” Cooling Solutions Serving HPC Centers• Reduced Cost, Both CAPEX and OPEX– Reduce or eliminate the need for traditionalchillers.
• No chillers, no ozone-depleting refrigerants (GREEN)– Oak Ridge calculates an annualized PUE for

Oak Ridge, TNair--cooled devices of no better than 1.4(ASHRAE Zone 4A – Mixed Humid)– HPC power budgets continue to grow – Summit has a design point for>12MW (HPC-only). Minimizing PUE/ITUE is critical to the budget.

• Easier, more reliable design– Design is reduced to pumps, evaporative cooling, heat exchangers.
– Traditional chilled water may not be necessary at all (NREL, NCAR, et al)
4

OLCF Facilities Supporting Summit• Titan – 9MW @ heavyload• Sitting on 250
pounds/ft2 raised floor• Uses 42F water andspecial CDUs (XDPs)• Summit – 256
compute cabinets
on-slab• 100% room-neutraldesign uses RDHX
• 20MW warm-watercooling plant usingcentralized
CDU/secondaryloop5

21°C/20MW/7700 ton Facility System Design System• Summit– Demand: 3.3 (idle)-11.5MW;• Secondary Loop
– Supply 3300GPM (12,500liters/min) @ 21°C; Return @ 2933°C– CPUs and GPUs use cold plates– DIMMs and parasitic loads use
RDHX– Storage and Network use RDHX

6

Medium Water Temperature Cooling Design
Primary Loop uses Evaporative
Cooling Towers (~80% of thehours of the year)
When the MTW RETURN is above
the 21C setpoint, use a second
set of Trim HX (with 5.5C on theother side) to drive MTW to the21C setpoint.
The need for the trim-loop is
about 20% of the hours in theyear, and can ramp 0-100% tomeet the setpoint back toSummit
A New Challenge – Titan androtated off-line, so there is no
significant load on theChillers[1-5]. NOAA moved tothe warm water loop (~500GPMdemand)
New primarycooling loop
7

Existing chilled water cooling loop

8

9

Benefits of ORNL’s Warm-Water (21°C) Mechanical Plant
Warm Water

Integrated System/FacilityOperation• Total Capacity to manage 20MW of IT• Integration with the PLC allows us to tune

load

• Warm Water allows annualized PUE of 1.1– For each ~$1M cost per MW-year for

consumption on Summit;– A corresponding ~$100k cost per MW-year

for waste heat management• Titan was $6-7M/year plus $1.8-$2M toremove the waste heat

• Summit will have a similar “power bill”,

but closer to $500k annually for wasteheat.

10water pressure and flow

– Better delta(t); less pumping energy• Integration with IBM’sOpenBMC provides
information necessary toprotect ~37K CPUs andGPUs from inadequate
flow across the cold plates• Integration with the scheduler allows us

to correlate power and temperature
data with individual applications

Summit’s Power Demand, Aug 2018– Aug 2019
Early Access, Gordon Bell, Transition to Operations
Acceptance Testing
Full Production11

2018/08/01
2018/08/11
2018/08/22
2018/09/02
2018/09/12
2018/09/23
2018/10/04
2018/10/14
2018/10/25
2018/11/05
2018/11/16
2018/11/26
2018/12/07
2018/12/18
2018/12/28
2019/01/08
2019/01/19
2019/01/30
2019/02/09
2019/02/20
2019/03/03
2019/03/13
2019/03/24
2019/04/04
2019/04/15
2019/04/25
2019/05/06
2019/05/17
2019/05/27
2019/06/07
2019/06/18
2019/06/282019/07/09
2019/07/20
2019/07/31
2019/08/10
2019/08/21

Cooling (kWcm)
14000

12000

10000

128000

Seasonal
Transition(Summer toFall)
Seasonal
Transition (toSummer)

Winter/Spring: 100%
Evaporative Cooling
1.15

6000

4000
1.10

2000
1.05

01.00

Date

CHW
CTW
1 Week PUE (kW)PUE

Detailed Analysis of Summit’s Annual PUE
Cooling Source August 1 2018-August 31 2019 With 1 Week PUE
1.30

1.25

1.20

Summit’s RM Challenge – Data Volume•

Data Streams Include:•IBM OpenBMC framework (99 metrics/node/second x 4608 nodes =~460,000/sec) - OOB•

IBM LSF jobs data for running applications (~10sec update interval)•NOAA weather/wet bulb for Oak Ridge area (continuous, external)•Programmable Logic Circuit (PLC) water flow (continuous, protectedas part of BAS)TOTAL ~470,000 metrics per second available for real-time analytics

13

13

Sample from Grafana Dashboard (live version shown)

14

MTW Performance

2018:Supply 70°F & ~4500GPM
2019:Supply 71°F & ~3300GPM

Wet Bulb
PUE Improves
Total Energy,
Mechanical Plant

Summit IT Load (kW-h)(Cumulative) Increases

Energy usedby ChilledWaterYear over Year, 2018 to 2019:• IT Demand (kW-h) is up 19%• CHW use is 39% LESS than in 2018
• The one-degree adjustment of supply
temperature provided a savings of $40,000
15Summit (System) Impact:• No substantive impact to CPU or GPU
operating conditions (junctiontemperatures)• One degree increase in room
temperature (worst case about 72.5° F)

Summit Power/Jobs Every Second

Click toPlay
Movie

16

Summit Temperature/Jobs Every Second

Click toPlayMovie

17

18