NREL is a national laboratory of the U.S. Department of Energy

Office of Energy Efficiency & Renewable Energy

Operated by the Alliance for Sustainable Energy, LLC

This report is available at no cost from the National Renewable Energy

Laboratory (NREL) at www.nrel.gov/publications.

Contract No. DE-AC36-08GO28308

Energy Performance Testing 

of Asetek’s RackCDU System 

at NREL’s High Performance 

Computing Data Center

David Sickinger, Otto Van Geet, and 

Chris Ravenscroft

National Renewable Energy Laboratory

Technical Report

NREL/TP-2C00-62905

November 2014
NREL is a national laboratory of the U.S. Department of Energy

Office of Energy Efficiency & Renewable Energy

Operated by the Alliance for Sustainable Energy, LLC

This report is available at no cost from the National Renewable Energy

Laboratory (NREL) at www.nrel.gov/publications.

Contract No. DE-AC36-08GO28308

National Renewable Energy Laboratory

15013 Denver West Parkway

Golden, CO 80401

303-275-3000 • www.nrel.gov

Energy Performance Testing 

of Asetek’s RackCDU System 

at NREL’s High Performance 

Computing Data Center

David Sickinger, Otto Van Geet, and 

Chris Ravenscroft

National Renewable Energy Laboratory

Prepared under Task No. WFT9.1000

Technical Report

NREL/TP-2C00-62905

November 2014
NOTICE

This report was prepared as an account of work sponsored by an agency of the United States government. 

Neither the United States government nor any agency thereof, nor any of their employees, makes any warranty, 

express or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of 

any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately 

owned rights. Reference herein to any specific commercial product, process, or service by trade name, 

trademark, manufacturer, or otherwise does not necessarily constitute or imply its endorsement, recommendation, 

or favoring by the United States government or any agency thereof. The views and opinions of authors 

expressed herein do not necessarily state or reflect those of the United States government or any agency thereof.

This report is available at no cost from the National Renewable Energy

Laboratory (NREL) at www.nrel.gov/publications.

Available electronically at http://www.osti.gov/scitech

Available for a processing fee to U.S. Department of Energy

and its contractors, in paper, from:

U.S. Department of Energy

Office of Scientific and Technical Information

P.O. Box 62

Oak Ridge, TN 37831-0062

phone: 865.576.8401

fax: 865.576.5728

email: mailto:reports@adonis.osti.gov

Available for sale to the public, in paper, from:

U.S. Department of Commerce

National Technical Information Service

5285 Port Royal Road

Springfield, VA 22161

phone: 800.553.6847

fax: 703.605.6900

email: orders@ntis.fedworld.gov

online ordering: http://www.ntis.gov/help/ordermethods.aspx

Cover Photos: (left to right) photo by Pat Corkery, NREL 16416, photo from SunEdison, NREL 17423, photo by Pat Corkery, NREL 

16560, photo by Dennis Schroeder, NREL 17613, photo by Dean Armstrong, NREL 17436, photo by Pat Corkery, NREL 17721.

NREL prints on paper that contains recycled content.
iii

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Abstract

In this study, we report on the first tests of Asetek's RackCDU direct-to-chip liquid cooling 

system for servers at NREL's ESIF data center. The system was simple to install on the existing 

servers and integrated directly into the data center's existing hydronics system. The focus of this 

study was to explore the total cooling energy savings and potential for waste-heat recovery of 

this warm-water liquid cooling system. RackCDU captured up to 64% of server heat into the 

liquid stream at an outlet temperature of 89°F, and 48% at outlet temperatures approaching 

100°F. This system was designed to capture heat from the CPUs only, indicating a potential for 

increased heat capture if memory cooling was included. Reduced temperatures inside the servers 

caused all fans to reduce power to the lowest possible BIOS setting, indicating further energy 

savings potential if additional fan control is included. Preliminary studies manually reducing fan

speed (and even removing fans) validated this potential savings but could not be optimized for

these working servers. The Asetek direct-to-chip liquid cooling system has been in operation 

with users for 16 months with no necessary maintenance and no leaks.
iv

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Acknowledgments

This document was prepared by the National Renewable Energy Laboratory (NREL) as the final 

report for the Cooperative Research and Development Agreement (CRADA) titled “NREL 

Energy Performance Testing of Asetek’s RackCDU System on a Single Rack of Twin Blade 

Servers,” contract number CRD-13-518.

NREL would like to thank key partner Asetek (San Jose, California) for providing the RackCDU 

liquid cooling system and technical assistance in the rack retrofit, and for covering the pipe runs 

from the facility system to the RackCDU. Special appreciation goes to Steve Branton, Anderson 

Tsai, Larry Seibold, David Tilton, and Steve Empedocles.

Another key partner was Aspen Systems (Denver, Colorado), who performed the rack retrofit 

and provided software programming support in running tests and collecting data from a variety 

of sources. Special appreciation goes to Jaime McIntyre, Justin Saunders, Ben Connell, and Erin 

Ward.

Project team members included Matt Bidwell, Ryan Elmore, Ian Metzger, and Kevin Regimbal 

from NREL, along with Dave Martinez from Sandia National Laboratories—all of whom 

provided valuable input and constructive feedback along the way.

We would like to thank Devonie McCamey of NREL for editing this document and Joshua

Bauer of NREL for graphic design support, along with the NREL researchers utilizing the 

compute rack that was retrofitted.
v

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

List of Acronyms

AHU air handling unit

BIOS basic input/output system

Btu British thermal unit

CDU cooling distribution unit

cfm cubic feet per minute

CPU central processing unit

CRADA Cooperative Research and Development Agreement

DoD U.S. Department of Defense

DOE U.S. Department of Energy

ERW energy recovery water

ESIF Energy Systems Integration Facility

ESTCP Environmental Security Technology Certification Program

FEMP Federal Energy Management Program

gpm gallons per minute

HPC High Performance Computing

IPMI Intelligent Platform Management Interface

kW kilowatt (one thousand watts)

MW megawatt (one million watts)

NREL National Renewable Energy Laboratory

OEM original equipment manufacturer

PDU power distribution unit

PUE power usage effectiveness

rpm revolutions per minute

W watt
vi

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Executive Summary

The ultimate goal of data center design is to be able to cool data center servers for “free” in any 

climate and eliminate the need for expensive and energy-demanding mechanical chillers, saving 

both capital and operating expenses. Warm-water direct-to-chip liquid cooling is capable of 

eliminating chillers in any climate and offers opportunities to use waste heat generated from 

servers as a heat source for buildings, domestic hot water, and other applications that require 

heat. The project covered in this report entailed retrofitting a compute rack at the National 

Renewable Energy Laboratory (NREL) from an air-cooled configuration to using Asetek’s 

RackCDU warm-water liquid cooling system in order to test and validate its cooling 

performance and long-term reliability. Direct measurements were made to compare and contrast 

energy consumption and performance both before and after RackCDU installation.

The compute rack chosen for this retrofit project belongs to the NREL Skynet high performance 

computing (HPC) cluster production system that was moved from a legacy data center to the 

Energy Systems Integration Facility (ESIF) HPC data center in March of 2013. The ESIF HPC 

data center itself is designed as a showcase facility for energy efficiency, incorporating the 

newest and most efficient designs and technologies available. It was designed to achieve an 

annualized average power usage effectiveness (PUE) rating of 1.06 or better. Additionally, waste 

heat from the HPC systems is captured and used as the primary heat source in the building's 

office and laboratory space. The Skynet compute hardware consists of fourteen 2U Twin/Twin 

Supermicro servers housing 56 Supermicro 2U Twin2 nodes, with rack power under full load 

around 15 kW (51,200 Btu/hr).

This work was performed under a Cooperative Research and Development Agreement 

(CRADA) between NREL and Asetek. Key partners included Asetek, who provided the 

RackCDU along with technical assistance, and Aspen Systems, who performed the rack retrofit 

and provided software programming support for running tests. Measurements under a fixed 

artificial load condition at 50% and 100% of the server capacity (running LINPACK) were 

recorded before and after the retrofit to determine the quantities of energy being dissipated to air 

and water in order to compare and contrast energy consumption and performance. Measurements 

included inlet/outlet temperatures along with flow rates for both air and water through the 

system, along with the power going into the rack. Appendix E contains the test condition results 

in table form after a semi-steady state was reached; values include: total heat removed by air and 

liquid, energy recovery to water, server central processing unit (CPU) temperatures, and server 

fan speeds.

Within a server, the CPUs and memory generate most of the heat load. Standard data center 

cooling techniques involve placing heat sinks on the high-heat-generating components and using 

internal fans to draw cold air across these heat sinks and other electrical board components to 

keep all the IT hardware components within their acceptable operating temperatures. Solutions 

such as those offered by Asetek’s RackCDU system bring liquid coolant directly to the hottest 

components to remove the majority of heat being generated, thus increasing the overall 

efficiency—as liquid cooling as a heat-exchange medium is three orders of magnitude more 

efficient than air. Asetek pump/cold plate assemblies were applied to the Skynet compute nodes 

during the retrofit. Fans are still utilized to cool the other electrical components, but there is 

opportunity to reduce fan speed or even eliminate some of the server fans. This project had two 
vii

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

distinct phases involving two versions of the Asetek RackCDU (V1.0 and V1.1), and during the 

second phase an aggressive attempt was made to actually disable operation of some of the server 

fans.

After the Skynet system was relocated from a legacy data center to ESIF, the fan speed control 

mode on the servers was changed to the most energy-efficient mode the BIOS allowed (set to 

Energy Savings mode from Performance). Testing in the air-cooled state was the only time the 

airflow rate noticeably changed, as after the retrofit the airflow remained at or near the minimum 

allowed through the server BIOS settings. The CPU temperatures ran noticeably higher under 

typical workloads when the Skynet rack was in this original air-cooled configuration.

The ESIF data center is provisioned to primarily support liquid cooling solutions, so only a few 

days were required prior to the retrofit to add the additional facility water pipe runs required by 

the RackCDU—which was done without any impact to normal data center operations.

Observations for the first phase of the project, with RackCDU version 1.0:

• Installation was very straightforward: The RackCDU bolt-on rack extension attaches to 

the standard rack after connecting supply/return facility lines; retrofit of the 14 servers 

took a three-person team 40 minutes each (10 minutes per dual-processor node) to 

convert and run a test harness (<10 hours for rack).

o The 10.5 inch (267 mm) rack extension houses a liquid-to-liquid heat exchanger.

o The server retrofit replaces CPU heat sinks with redundant pump/cold plate units 

that sit atop and collect heat from the CPUs as server cooling water circulates 

through.

• 44% to 48% energy recovery was observed at typical workloads, with water returning 

around 95°F (35°C) that the ESIF Energy Recovery Water (ERW) system can use as a 

building heat source.

o Facility water supply was 62°F (16.7°C), and the RackCDU control parameter for 

facility return setpoint was 100°F (37.8°C), which adjusts facility flow rate.

• With all the server fans running in the Energy Savings mode, there were no noticeable fan 

energy savings—so no significant server power savings.

• The retrofit of the compute rack with Asetek RackCDU was commissioned by Aspen 

Systems on June 10, 2013.

Opportunities were identified with Asetek and Aspen Systems to improve performance by 

reducing fan energy and thermal losses. As a result, another retrofit of Skynet rack occurred, 

which also included a newer version of the RackCDU unit that retained the server pump/cold 

plate units.

Observations for the second phase of the project, with new RackCDU version 1.1:

• The RackCDU bolt-on rack extension was replaced; again, this was straightforward, as 

cooling liquid moves between the servers and rack extension utilizing hoses with dripless 

quick connects.
viii

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

• An aggressive attempt was made to reduce fan energy, with an operational test period 

involving seven servers each operating with half of the server fans disabled starting in 

March 2014.

o This configuration passed a short 100% load condition running LINPACK, so it 

moved into an operational test period.

o During operational testing, server air-cooled components caused the system to 

become unstable at times—so it was reverted back to running all the server fans in 

Energy Savings mode as the final system configuration state in May 2014.

• 64% energy recovery was observed at typical workloads in the final system 

configuration, using a higher facility water flow rate but lower return facility water 

temperature of 89°F (31.7°C).

o Control parameters can be set for the RackCDU that include a minimum allowed 

for facility flow through rack extension; the default value was set higher during

testing.

o Facility water supply and facility return setpoint were the same as before.

The Skynet HPC cluster is a production system, so the number and duration of outages made 

available for the RackCDU installation and testing at constant load conditions was limited—and

the ESIF facility environmental conditions could not be altered for any testing (like the supply 

water temperatures). During testing in the first phase of the project, some of the test conditions 

led to water energy recovery in the ~30% range—which occurred when trying to drive the ERW

temperature well above 95°F (35°C) into the 110°F (43°C) range for use as a heat source for 

buildings, domestic hot water, and other applications requiring heat.

The opportunity to reduce server fan energy was very appealing to NREL, and the aggressive 

attempt made to lower fan energy by disabling some of the fans showed potential. Ideally an 

original equipment manufacturer (OEM) would include fan speed settings for when CPU and

memory components are liquid cooled or allow custom settings. If the Skynet compute hardware 

had consisted of more than one similarly configured rack, perhaps more test opportunities would 

have been possible to explore different combinations to reduce server fan energy on a single rack 

—then after a successful operational test period, that solution could be rolled out to the 

remaining racks.

The higher flow rates tested with the new version of RackCDU correspond to a greater water 

energy recovery percentage; this matches observations made in a report by Lawrence Berkeley 

National Laboratory (LBNL) on a similar demonstration project. The study by LBNL included 

varying the supply water temperature and water flow rates at three different power levels (idle, 

50%, and 100% load conditions) on servers that had Asetek pump/cold plate assemblies applied 

to both the CPU and memory components:

• “For a given supply water temperature, a higher flow rate corresponds to a greater 

fraction of heat captured. This behavior was expected because the water flow rate on the 

server side was constant, with the result that higher flow on the low-temperature side in 

turn dropped the temperature on the server side, thereby extracting a higher fraction of 

the heat from the direct-cooled components.” (Coles and Greenberg, 2014)
ix

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

When the 14 Skynet 2U servers were originally installed in a legacy data center, they had to be 

spread across two racks based on available air cooling options for that room. Using warm-water 

direct-to-chip liquid cooling solutions such as Asetek’s RackCDU to reject more than half the 

heat directly to liquid would allow density to be doubled in the case of the Skynet hardware. It 

might also allow for the server fan speed control mode to be changed to a more energy-efficient 

setting for additional savings.

With more than a year (going on 16 months) of the RackCDU system being in service on a 

production system, it has proven to be a very low-maintenance cooling system with high

reliability, as no maintenance has been required during this time period and there have been no 

leaks. The RackCDU system was easy to integrate as a retrofit-in-place option.
x

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Table of Contents

1 Introduction........................................................................................................................................... 1

1.1 Project Background....................................................................................................................... 1

1.2 HPC Data Center at NREL............................................................................................................ 1

1.3 Participants.................................................................................................................................... 2

2 Methods................................................................................................................................................. 2

2.1 Compute Rack – IT Equipment (Servers)..................................................................................... 4

2.1.1 Server Data....................................................................................................................... 5

2.2 Air Configuration Testing (Prior to RackCDU Installation)......................................................... 5

2.2.1 Air Sensor Data ................................................................................................................ 6

2.3 Retrofit: Converting from Air to Mixed Liquid- / Air-Cooled System......................................... 7

2.3.1 Facility Preparation .......................................................................................................... 8

2.3.2 Retrofit of Skynet Rack.................................................................................................... 9

2.4 Liquid Configuration Testing...................................................................................................... 11

2.4.1 Liquid Sensor Data......................................................................................................... 11

3 Test Results ........................................................................................................................................ 12

3.1 Air-Cooled Configuration ........................................................................................................... 12

3.2 Liquid-Cooled Configuration...................................................................................................... 13

3.3 Second Retrofit Results............................................................................................................... 16

3.3.1 Aggressive Reduction in Fan Energy Attempt............................................................... 17

3.3.2 Final Configuration with Updated RackCDU................................................................ 18

4 Conclusions ........................................................................................................................................ 19

4.1 Observations for the First Phase of the Project ........................................................................... 19

4.2 Observations for the Second Phase of the Project....................................................................... 19

4.3 General Observations.................................................................................................................. 20

5 Suggestions for Further Work........................................................................................................... 20

References................................................................................................................................................. 21

Appendix A: Air-Cooled Data................................................................................................................... 22

Appendix B: Liquid-Cooled Data............................................................................................................. 24

Appendix C: Second Retrofit Attempt .................................................................................................... 26

Appendix D: Testing – Lessons Learned ............................................................................................... 28

Appendix E: Tables from Section 3 Grouped Together........................................................................ 30
xi 

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

List of Figures

Figure 1. Asetek pump/cold plate units and RackCDU. ............................................................................... 3

Figure 2. Supermicro 2U Twin/Twin server. Images from Super Micro Computer, Inc.............................. 4

Figure 3. Rack layout—legacy data center vs. ESIF data center. ................................................................. 5

Figure 4. Ganglia load for August 8, 2013, test period (showing nodes 29–32). ......................................... 5

Figure 5. Air capture hood............................................................................................................................ 6

Figure 6. Test setup....................................................................................................................................... 7

Figure 7. Building-level system diagram...................................................................................................... 8

Figure 8. Line drawing of facility pipe additions to support Asetek RackCDU........................................... 9

Figure 9. Server fan configuration during aggressive second retrofit attempt............................................ 17

Figure 10. Temperature sensor arrangement (back of rack) and chassis back view. Images from Super 

Micro Computer, Inc.............................................................................................................. 28

Figure 11. Thermocouple array, back of rack air temperatures for March 16, 2013. ................................. 29

Figure 12. Thermocouple array, back of rack air temperatures for August 8, 2013. .................................. 29

List of Tables

Table 1. Mean Energy Flow Rates for May 16, 2013, Test Conditions...................................................... 13

Table 2. Mean Server Data for May 16, 2013, Test Conditions................................................................. 13

Table 3. Mean Energy Flow Rates for August 8, 2013, Test Conditions ................................................... 14

Table 4. Mean Energy Flow Rates using Backup Reference Flow Meter for August 8, 2013................... 15

Table 5. Mean Server Data for August 8, 2013, Test Conditions............................................................... 15

Table 6. Mean Energy Flow Rates, Two States While Rack in Production (Hour Timeframe)................. 16

Table 7. Mean Server Data, Two States While Rack in Production (Hour Timeframe)............................. 16

Table 8. Mean Energy Flow Rates for March 27, 2014, Test Conditions .................................................. 18

Table 9. Mean Server Data for March 27, 2014, Test Conditions.............................................................. 18

Table 10. Mean Energy Flow Rates for May 7, 2014................................................................................. 18

Table 11. Mean Server Data for May 7, 2014 ............................................................................................ 19
1

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

1 Introduction

Today's typical data centers consist of air-cooled servers that require 80°F (26.7°C) at the face of 

the server (ASHRAE 2011 Thermal Guidelines for Data Processing Environments), even though 

the CPUs and other components in the server can run at 150°F+ (65.6°C+). This is because air is 

not very dense and therefore has limited heat transfer abilities. Water as a heat-exchange medium 

is three orders of magnitude more efficient than air, and getting the heat exchange close to where 

the heat is generated is most efficient.

The ultimate goal of data center design is to be able to cool data center servers for “free” in any 

climate and eliminate the need for expensive and energy-demanding mechanical chillers, saving 

both capital and operating expenses. Warm-water direct-to-chip liquid cooling is capable of 

eliminating chillers in any climate. A secondary benefit of warm-water cooling is that the 

110°F+ (43.3°C+) water leaving the system can be used as a heat source for buildings, domestic 

hot water, and other applications that require heat.

1.1 Project Background

This project entailed retrofitting a compute rack at the National Renewable Energy Laboratory 

(NREL) from an air-cooled configuration to using Asetek’s RackCDU warm-water liquid 

cooling system in order to test and validate its cooling performance and long-term reliability. 

Direct measurements were made to compare and contrast energy consumption and performance 

both before and after RackCDU installation. The scope of this project was to demonstrate and 

provide feedback on the Asetek RackCDU system.

This work was performed under a Cooperative Research and Development Agreement 

(CRADA) between NREL and Asetek with the purpose: “To measure the energy savings, 

performance and reliability of a warm-water direct-to-chip liquid cooling retrofit solution for 

data centers.”

NREL would like to thank the Department of Energy (DOE) Federal Energy Management 

Program (FEMP) for providing funding for a portion of this work. NREL has also partnered with 

Asetek on a Department of Defense (DoD) Environmental Security Technology Certification 

Program (ESTCP) project to test the RackCDU in a DoD facility in the Southeastern United 

States. The results of the work at NREL have been incorporated in the ESTCP project.

1.2 HPC Data Center at NREL

NREL's Energy System Integration Facility (ESIF) high performance computing (HPC) data 

center itself is designed as a showcase facility for energy efficiency, incorporating the newest 

and most efficient designs and technologies available. It was designed to achieve an annualized 

average power usage effectiveness (PUE) rating of 1.06 or better. Additionally, waste heat from 

the HPC systems is captured and used as the primary heat source in the building's office and 

laboratory space. Together, the energy efficiency and energy reuse features combine to make the 

ESIF data center one of the most energy-efficient data centers in the world. The ESIF HPC data 

center is designed to support power and cooling infrastructure up to a maximum of 10 MW

(34,120,000 Btu/hr). Initially, it has been equipped (with generators, power distribution panels, 

cooling towers, fans, etc.) to power and cool up to 2 MW (6,824,000 Btu/hr) of electrical load. 

While both air- and liquid-based cooling solutions can be accommodated, the ESIF data center is 
2

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

currently provisioned to air-cool up to a maximum of 200 kW (682,400 Btu/hr) of the total heat 

load generated by HPC systems—so there is 1.8 MW (6,142,000 Btu/hr) available for systems 

that dissipate heat directly to liquid. Water delivered to HPC systems can vary between 60°F and 

72°F (15.6°C and 22.2°C), and typically has been 62°F (16.7°C) to support some of the current 

air-cooled systems. Desired water temperatures leaving HPC systems is 95°F+ (35°C+) to 

maximize the use of waste heat within the ESIF building.

The compute rack chosen for this retrofit project belongs to the NREL Skynet HPC cluster 

system that was originally installed in NREL’s legacy data center in early 2012. That data center 

operated at an average PUE of 2.28, which is consistent with typical data centers (Sheppy et al. 

2011). While the system was located at this legacy data center, the compute hardware consisting 

of fourteen 2U Twin/Twin Supermicro servers had to be spread across two racks based on 

available cooling options for that room. When the Skynet system was relocated to the ESIF data 

center in March of 2013, the fourteen 2U servers were reconfigured into one rack and placed in a 

row containing other air-cooled systems that utilizes a full hot-aisle containment strategy to 

eliminate mixing of hot exhaust air with the cool supply air. The rack power under full load is 

around 15 kW (51,200 Btu/hr), while idle is around 7 kW (23,900 Btu/hr).

1.3 Participants

The NREL Skynet HPC cluster is a production system, and the user community was willing to 

accept a few system outages to accommodate the rack conversion and testing when the Skynet 

system was relocated to the ESIF data center. Asetek provided the RackCDU cooling system, 

including piping modifications to tie into NREL’s facility water energy recovery loop along with 

the retrofit kits for fourteen 2U Twin/Twin Supermicro servers and technical support. Aspen 

Systems performed the rack retrofit (alongside Asetek staff) and provided software programming 

support to run tests and collect data from variety of sources. Aspen Systems also handled the 

system move from NREL’s legacy data center to the new ESIF data center. 

2 Methods

The main objective of this project was to determine the quantities of energy being dissipated to 

air and water in order to compare and contrast energy consumption and performance both before 

and after RackCDU installation. In order to accomplish this, measurements were taken under 

constant load conditions (50% and 100%) using the LINPACK benchmark before and after 

installation of the RackCDU. Measurements were also taken under the same load conditions at 

different water return temperatures (controlled by adjusting the facility water flow rate through a

RackCDU control parameter). Measurements included air inlet/outlet temperatures and airflow

through the rack; the same was done when liquid cooling was added (liquid temperatures and 

flow rates were measured), as well as power to the rack.

It is important to note that the Skynet HPC cluster is a production system, so the number and 

duration of outages made available for the RackCDU installation and testing at constant load 

conditions were limited. Also of note is that the facility conditions could not be altered for any 

testing, and the facility supply water has typically been at the low end of the range, around 62°F 

(16.7°C). The best attempt was made during testing to reach a semi-steady state for estimating

energy recovered using facility-caliber instrumentation. The results are presented in table and 
3

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

graphical form to help identify trends that could be confirmed later with additional testing, 

ideally using laboratory-grade instrumentation.

Within a server, the CPUs and memory generate most of the data center heat load. Standard data 

center cooling techniques involve placing heat sinks on certain high-heat-generating components 

and using internal fans to draw cold air across these heat sinks and other electrical board 

components to keep all the IT hardware components within their acceptable operating 

temperatures. Solutions such as those offered by Asetek’s RackCDU system bring liquid coolant 

directly to the hottest components to remove the majority of heat being generated, thus 

increasing the overall efficiency—as liquid cooling as a heat-exchange medium is three orders of 

magnitude more efficient than air. Fans are still utilized to cool the other electrical components, 

but there is opportunity to reduce fan speed or even get eliminate some of the server fans.

The Asetek RackCDU system includes a 10.5 inch (267 mm) bolt-on rack extension that houses 

a liquid-to-liquid heat exchanger (HEX). Facility supply and return water runs to one side of this 

HEX unit, in this case connected to the ESIF ERW loop that is described later in Section 2.3. 

The other side of the HEX connects to the individual servers using hoses with dripless quick 

connects to move server cooling liquid between the RackCDU and servers. Asetek supplied the 

liquid on the server cooling side of the RackCDU, which is mainly water with some anticorrosion additives. The other main components of the RackCDU system are the redundant 

pump/cold plate units that sit atop and collect heat from the CPUs (replacing the CPU heat sink) 

as server cooling water is circulated through; there are also cold plates for memory, but these 

were not incorporated into the Skynet servers that were retrofitted.

Figure 1. Asetek pump/cold plate units and RackCDU.
4

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

The rest of this chapter focuses on:

• Compute rack IT equipment and the servers that were retrofitted

• Air configuration testing prior to RackCDU installation

• Retrofit: Converting from air to mixed liquid-/air-cooled system

• Liquid configuration testing.

2.1 Compute Rack – IT Equipment (Servers)

The compute rack of the NREL Skynet HPC cluster system is the hardware that was retrofitted 

with the Asetek RackCDU. It consists of fourteen 2U Twin/Twin Supermicro servers housing 56 

Supermicro nodes. The Twin/Twin servers are configured with a Supermicro Chassis integrated 

with four Supermicro Motherboards. The Motherboards are dual 1366-pin LGA Socket boards

for the Intel Xeon 5600/5500 Series. The Chassis includes a 1400W (4,800 Btu/hr) Gold Level 

High-efficiency Redundant (1+1) Power Supply, and four 80 mm (3.1 inch) PWM cooling fans. 

Each Compute Node is configured with two Intel Westmere X5667, 3.06GHz Quad-Core 95-

Watt (324 Btu/hr) processors, and 24GB of DDR3 1333MHz ECC Registered memory for 3GB 

of memory per core (4GB x 6 per node). On paper, the power requirement for each of the servers 

is listed as 1.12 kW (3,800 Btu/hr). With eight 95W (324 Btu/hr) processors per server, the CPU 

power accounts for about 68% of the power requirements. 

Figure 2. Supermicro 2U Twin/Twin server. Images from Super Micro Computer, Inc.

When the fourteen 2U servers were originally installed in NREL’s legacy data center, they had to 

be spread across two racks based on available cooling options for that room that involved the use 

of in-row cooling systems. When the Skynet system was relocated to the ESIF data center in 

March of 2013, the fourteen 2U servers were reconfigured into one rack (filling 28U out of a 

42U rack) and placed in a row that utilized a full hot-aisle containment strategy to eliminate 

mixing of hot exhaust air with the cool supply air. The BIOS settings for the server fan speeds 

were also changed to the Energy Efficient mode after some initial test runs revealed the server 

fan speeds didn’t change while operating under different loads when set to the Performance

mode (the fan setting used in NREL’s legacy data center). All tests reported in Section 3 were 

conducted with the same Energy Efficient BIOS setting in place for the server fans.
5

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Figure 3. Rack layout—legacy data center vs. ESIF data center.

2.1.1 Server Data

On the test days reported in Chapter 3, the ‘lm-sensors’ package (a Linux hardware health 

monitoring package) was used to record CPU temperatures, and the server fan speeds were 

recorded using Supermicro SuperDoctor II. Both values were written to individual text files for 

each of the 56 nodes at 15-second intervals.

The system-monitoring tool Ganglia was also used to view CPU load averages on each of the 

nodes to look at the number of processes running. Figure 4 shows nodes 29–32 for August 8, 

2013 (typical of all 56 nodes). As each node has two quad-core CPUs, full utilization is 8 when 

running a full 100% constant load using the LINPACK benchmark (50% constant load is 4). 

Under normal rack operations, there are occasionally times when the CPU load average might 

report 9 or 10 over a few nodes that reflect heavy network activity (although the system usually 

remains stable during these times).

Figure 4. Ganglia load for August 8, 2013, test period (showing nodes 29–32).

The power measurement was measured at the rack level through the two rack Power Distribution 

Units (PDUs) that only serve the Skynet compute rack (15-second sampling).

2.2 Air Configuration Testing (Prior to RackCDU Installation)

In order to gain a baseline, prior to conversion to liquid cooling, the energy transfer to air had to 

be measured. To conduct this measurement, a capture hood was fabricated to isolate and measure 

the air passing through the Skynet rack. The capture hood was designed to hold two differentialpressure airflow trees. Each tree is designed to measure the flow rate for a 1 ft2 area. The capture 
6

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

hood is designed to hold two trees at the inlet, to measure the volume of air moving through the 

compute rack.

Figure 5. Air capture hood.

To complete the information needed to calculate energy capture, the inlet and discharge 

temperatures had to be measured. From this information, the energy transfer was calculated 

using the following equation:

(a) Q = Cp*m*∆T

Q = energy flow rate (Btu/hr)

Cp = specific heat (Btu/lbm * °F)

m = mass flow rate (lbm/hr)

∆T = Temperature (°F)

To find the mass of air, the density has to be used to convert from cubic feet of air to pounds. 

Because the NREL campus is not at sea level, but approximately 5,900 feet above sea level, the 

change in air density must be accounted for. At sea level, the density of air is equal to 0.075 

lbm/ft3

, while at 5,900 feet the density of air drops to approximately 0.062 lb/ft3. The equation 

for energy flow rate then changes to the following:

(b) Q = Cp*ρ*60*cfm*∆T

ρ = density (lbm/ft3)

The specific heat, density, and the time conversion (60 minutes/hr) are all constants in this form 

of the equation. The constant, when multiplied through, is 1.08 when at sea level, and is 0.893 at 

5,900 feet of elevation. It is common practice to round this number to 0.9. With all the constants 

multiplied through, and only leaving in the variables, the equation is as follows:

(c) Qair = 0.9*cfm*∆T

2.2.1 Air Sensor Data

During testing under constant load conditions, the cardboard capture hood was taped to the front 

of the Skynet rack and the two differential pressure airflow tree readings were manually recorded

(utilizing a TSI Alnor balometer capture hood airflow tree, model EPT721). The supply air 
7

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

temperature to the rack was measured with three APC temperature sensors and recorded at oneminute intervals (sufficient as supply air temperature varied little during the individual test runs). 

These temperature sensors were spaced evenly in a low-middle-high pattern centered in the rack.

The air temperature exiting the rack was measured using nine thermocouples arranged in a 3x3 

array pattern, and a Fluke 2635A data logger recorded these readings at five-second intervals. 

Appendix D goes into more detail about lessons learned in the placement of these, but only the 

low-middle-high sensors in the center section of the rack were used for the mean rack exit 

temperature (Tout) values appearing in the tables in Section 3.

Figure 6. Test setup.

2.3 Retrofit: Converting from Air to Mixed Liquid- / Air-Cooled System

The ESIF data center is a warm water, non-mechanically cooled system (building-level system 

diagram shown in Figure 7) that was built with liquid-cooled HPC equipment in mind. To 

achieve this, a set of heat exchangers and cooling coils have been designed to transfer energy, or 

heat, from the data center to either a set of cooling towers, the building heating loop, or a 

combination of the two. While both air- and liquid-cooled data center equipment can be 

accommodated, the ESIF data center is currently provisioned to air-cool only up to 200 kW 

(682,400 Btu/hr) electrical load of data center equipment. The air-cooled racks are maintained by 

a closed-circuit air handling system that uses the ERW system to cool the air. The Skynet 

compute rack that was retrofitted with the Asetek RackCDU became a partially liquid-cooled 

and partially air-cooled hybrid system.
8

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Figure 7. Building-level system diagram.

2.3.1 Facility Preparation

The ERW system is located in a mechanical room directly below the data center and has a series 

of flanges (taps) to accommodate pipe runs to liquid-cooled racks or equipment in the data 

center. The Asetek conversion line drawing is expanded in Figure 8 to show the added pipe runs 

from the facility flanges to the Asetek RackCDU connection points, along with some sensor 

point locations (temperature, flow meter) for monitoring performance. This work was completed 

over the course of a few days prior to the rack retrofit, without any impact to normal data center 
9

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

operations (although additional pipe runs did require some indoor hot-work permits). With the 

modulating pressure independent control valve wide open, a maximum flow rate of 6.1 gpm 

(23.1 L/min) was recorded in the ERW return (ERWR) line that connects to the Asetek 

RackCDU.

Figure 8. Line drawing of facility pipe additions to support Asetek RackCDU.

2.3.2 Retrofit of Skynet Rack

The actual retrofit of the Skynet compute rack occurred over two days and was commissioned by 

Aspen Systems on June 10, 2013. Each of the 56 Supermicro compute nodes were processed to 

replace the CPU heatsinks with the Asetek pump/cold plate assemblies (internal cooling loops) 

which transport waste heat to the RackCDU through tubes that exit through an unused PCIe slot. 

The tubes are attached to the RackCDU using dripless quick connects that move cooling liquid 

between the RackCDU and servers. The RackCDU is mounted in a 10.5 inch (267 mm) rack 

extension that attaches to the back of the rack.

The major steps involved are as follows:

1. Install the RackCDU:

a. Connect the facility supply and return water lines to the rack extension.

b. Lift the rack extension into place and attach to the back of the rack.

c. Connect control valve wires to RackCDU control pack.

d. Connect power to RackCDU unit (plugs directly into a rack-mounted power 

strip).

e. Connect and configure network to RackCDU.

f. Adjust RackCDU alarm settings and desired facility temperature return.

2. Retrofit the 14 servers with Asetek pump/cold plate assemblies—40 minutes per server 

(10 minutes per dual-processor node) for three-person team (around 9 hours and 20 

minutes total):

a. Take the servers offline, disconnect power/network cables, and remove server 

from the rack.
10

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

b. Slide each of the four nodes out of the server.

c. Remove air shroud and two passive CPU heatsinks from each node to allow 

Asetek hardware installation, and wipe off the remaining thermal paste from the 

CPUs.

d. Route the tubes for the pump/cold plate assembly in each node, and secure the 

cold plates to the CPUs (cold plates include thermal paste pre-applied with 

removable clear plastic cover).

e. Slide each of the four nodes back into the server.

f. Connect server to a test harness assembly to verify that server powers up and that 

water is flowing through internal cooling loop.

Heatsinks

Air shroud (removed)

Asetek pump/cold plate 

Server fans (4x)

CPU-1

CPU-2
11

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

g. Install server back in rack, and connect power/network cables and the hoses with 

dripless quick connects that move server cooling liquid between the RackCDU 

and servers.

3. Test the rack under full simulated load using LINPACK.

2.4 Liquid Configuration Testing

Determining the energy going to water for cooling uses the same energy transfer equation (a), 

except with the constants needed to convert gallons per minute (gpm) of water into pounds per 

hour for the proper energy units. In this case, there is not a conversion for altitude on the density 

of water, because of the minimal effect. Using the same equation, the constants for water are:

Cp = 0.998 Btu/lbm-°F

ρ = 8.33 lbm/gal

These constants are placed back in equation (b) and cfm is replaced with gpm. When the 

numbers are calculated, the constant value is 498.8. It is common practice to round the constant 

to 500. The equation for energy going to water then becomes the following:

(d) Qwater = 500*gpm*∆T

As the retrofitted Skynet rack is a hybrid-cooled system, both equations (c) and (d) are combined 

to measure the total energy flow rate.

2.4.1 Liquid Sensor Data

The water flow rate on the facility loop was measured in two different locations. The first was 

with an internal flow meter (part of the RackCDU) that adjusts the facility control valve to 

regulate water return temperatures to the desired setting. As a backup reference and to help with 

verifying the flow rates, an Onicon F-1100 flow meter (designated “FM” in Figure 8) was also 

installed in the return piping of the facility water with readings recorded by Fluke 2635A data 

logger. Two thermocouples were also installed on the facility pipe addition to measure the 

facility water supply and return temperatures to the RackCDU. The RackCDU also records 

facility supply/return temperatures, and these data were used as a secondary reference.

Test station 

Four retrofitted nodes ready

to go back into a server
12

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

3 Test Results

The results are presented in the following fashion:

• Air-cooled configuration, which was tested May 16, 2013 before the rack retrofit

• Liquid-cooled configuration for the first phase of the project, which was tested on August 

8, 2013, after rack was retrofitted with the Asetek RackCDU version 1.0

o Data were also recorded during normal rack operation on July 19, 2013, and 

August 13, 2013.

• Second phase of the project, review of additional changes attempted to reduce fan energy 

and test on March 27, 2014, then final rack configuration implemented May 7, 2014

o This included installation of a new Asetek RackCDU version 1.1.

Every attempt was made to allow at least 30 minutes per test condition while watching data 

during the test run to identify when thermal and power values appeared to have little variation, 

and then identify during the analysis stage an appropriately sized window (10 to 11 minutes) for

calculating a semi-steady-state approximation of energy transfer that usually occurred towards 

the end of a test run. In addition to the constant load condition testing (50% and 100% using the 

LINPACK benchmark), data were collected at times during normal rack operations. Two dates 

were selected that allowed for a one-hour window over which to calculate a semi-steady-state 

approximation: July 19, 2013, and August 13, 2013.

Two tables are shown for each test date. The first shows the mean data values needed to calculate 

the total mean energy flow rate (Qtotal), which is compared against power going into the rack as 

recorded by the two PDUs. Ideally, these should match, but there are uncertainty and 

measurement errors, and these test conditions truly only reached a semi-steady state. The second 

table reports the mean server data values for CPU-1 (those located closest to the server fans), 

CPU-2, and the reported server fan speeds. Comments/discussion appear before and after the 

tables as appropriate. Appendix E provides these same tables grouped together for making quick 

comparisons.

3.1 Air-Cooled Configuration

An initial set of tests conducted on the air-cooled rack configuration with the capture hood 

revealed that the BIOS fan settings were set to run high, as the recorded airflow measurements 

taken while running LINPACK at 50% load and 100% load were nearly identical (1,209 cfm and 

1,223 cfm, respectively). The BIOS settings for these servers support four different fan speed 

control modes as follows:

1. Full Speed (FS)

2. Performance (PF)

3. Balanced (BL)

4. Energy Savings (ES).

The Skynet server fans were originally set to run in Performance mode when the equipment was 

installed in NREL’s legacy data center. After an initial set of tests in the ESIF data center on 
13

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

May 8, 2013, the BIOS fan speed settings were changed to Energy Savings mode on May 15, 

2013, where they have remained since. As a reference point, the Performance mode fan speed 

reported in the BIOS environment while a server was in an idle state was 7,200 rpm compared to 

4,000 rpm when the BIOS setting was changed to Energy Savings mode.

After all the server BIOS fan speed settings were changed to Energy Savings mode, there was a 

brief test window available on May 16, 2013, to run the LINPACK tests again while the rack 

was still in the air-cooled configuration.

In this setup, Qtotal = Qair, so only Equation (c) is needed to fill out Table 1.

Qair = 0.9*cfm*(Tout – Tin)

For May 16, 2013 tests:

• Mean inlet air temperature (Tin, air) remained steady at 69.1°F (20.6°C).

Table 1. Mean Energy Flow Rates for May 16, 2013, Test Conditions

Test Condition,

May 16, 2013

Air Rack 

Power,

Btu/hr

(kW)

Missing

Energy

Percent

Flow,

cfm

(L/min)

Tout,

°F

(°C)

Qair,

Btu/hr

(kW)

50% LINPACK,

9:21 – 9:31:59

786

(22,260)

117.6

(47.6)

34,309

(10.1)

35,418

(10.4)

3.1%

100% LINPACK,

9:47 – 9:57:59

968

(27,414)

123.0

(50.6)

46,958

(13.8)

47,804

(14.0)

1.7%

Note: Node 4 was powered off during the testing on May 16, 2013, as it required repairs, so only 

55 of the 56 nodes were operational during testing.

Table 2. Mean Server Data for May 16, 2013, Test Conditions

Test Condition,

May 16, 2013

CPU-1 Temp,

°F

(°C)

CPU-2 Temp,

°F

(°C)

Server Fan

Speeds,

rpm

50% LINPACK

9:21 – 9:31:59

125.0

(51.7)

134.9

(57.2)

4501

100% LINPACK

9:47 – 9:57:59

150.4

(65.8)

156.8

(69.3)

5711

The CPU temperatures in the air-cooled configuration are noticeably higher than those found 

later in the liquid-cooled configuration. This is also the only time the server fan speeds 

noticeably adjusted to the different test conditions.

Additional graphs related to air-cooled configuration tests appear in Appendix A.

3.2 Liquid-Cooled Configuration

The retrofit of the Skynet compute rack occurred over two days and was commissioned by 

Aspen Systems on June 10, 2013. The first full dedicated day of testing occurred on August 8, 
14

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

2013 (results appear in Tables 3 through 5). Data were also collected while the Skynet rack was 

in production on July 19, 2013, and August 13, 2013 (results appear in Tables 6 and 7).

Equations (c) and (d) are now combined for Qtotal with RackCDU installed when filling out the 

energy flow rate tables from this point forward.

Qtotal = Qair + Qwater = (0.9*cfm*(Tout – Tin)) + (500*gpm*(Tout – Tin))

The RackCDU control parameter for desired facility water return temperature was raised to see 

how the system would respond on the August 8, 2013, test day while Aspen System engineers 

were on-site to monitor the Skynet system hardware. Higher-temperature return water is desired 

when the waste heat from servers can also be used as a heating source in buildings like ESIF. If 

the waste heat is simply rejected, then having a higher percentage of server energy recovered to 

water is more desirable, as water is a more efficient heat-exchange medium than air. For all other 

RackCDU testing, the desired facility water return setpoint was left at 100°F (37.8°C).

For the August 8, 2013 tests:

• Mean inlet air temperature (Tin, air) remained steady at 67.8°F (19.9°C).

• Mean water facility supply temperature (Tin, water) remained steady at 62.3°F (16.8°C).

• Take note of the desired facility water return “Setpoint” settings for four test conditions:

o These changed from 100°F (37.8°C) to 120°F (48.9°C) and then 140°F (60.0°C).

Table 3. Mean Energy Flow Rates for August 8, 2013, Test Conditions

Test Condition,

August 8, 2013

Air Water Qtotal,

Btu/hr

(kW)

Rack 

Power,

Btu/hr

(kW)

Water

Energy

Recovery

Percent

Missing

Energy

Percent

Flow,

cfm

(L/min)

Tout,

°F

(°C)

Qair,

Btu/hr

(kW)

Flow,

gpm

(L/min)

Tout,

°F

(°C)

Qwater,

Btu/hr

(kW)

50% LINPACK,

Setpoint 100°F

9:40 – 9:50:59

694

(19,654)

98.9

(37.2)

19,431

(5.7)

1.08

(4.1)

90.3

(32.4)

16,427

(4.8)

35,858

(10.5)

36,237

(10.6)

45.3% 1.0%

50% LINPACK, 

Setpoint 120°F

12:50 – 13:00:59

694

(19,654)

102.7

(39.3)

21,749

(6.4)

0.53

(2.0)

105.8

(41.0)

12,145

(3.6)

33,894

(9.9)

36,408

(10.7)

33.4% 6.9%

100% LINPACK, 

Setpoint 120°F

14:40 – 14:50:59

703

(19,909)

107.4

(41.9)

25,200

(7.4)

0.61

(2.3)

118.8

(48.2)

17,949

(5.3)

43,150

(12.6)

47,770

(14.0)

37.6% 9.7%

100% LINPACK, 

Setpoint 140°F

16:40 – 16:50:59

703

(19,909)

107.1

(41.8)

24,960

(7.3)

0.64

(2.4)

118.1

(47.8)

18,707

(5.5)

43,667

(12.8)

48,009

(14.1)

39.0% 9.0%

The facility pipe additions to support the Asetek RackCDU were sized to accommodate more 

than one RackCDU depending how the hot-aisle containment row developed. The backup 

reference flow meter installed in the return piping was thus calibrated to measure up to 14 gpm

(product catalog sheet lists a minimum operating range flow rate of 0.8 gpm)—so the water flow 

rates reported in the tables are from the Asetek RackCDU flow meter, but Table 4 shows energy 

flow rates using the backup reference flow meter for the first test condition of August 8, 2013.
15

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Table 4. Mean Energy Flow Rates using Backup Reference Flow Meter for August 8, 2013

Test Condition,

August 8, 2013

Air Water Qtotal,

Btu/hr

(kW)

Rack 

Power,

Btu/hr

(kW)

Water

Energy

Recovery

Percent

Missing

Energy

Percent

Flow,

cfm

(L/min)

Tout,

°F

(°C)

Qair,

Btu/hr

(kW)

Flow,

gpm

(L/min)

Tout,

°F

(°C)

Qwater,

Btu/hr

(kW)

50% LINPACK,

Setpoint 100°F

9:40 – 9:50:59

694

(19,654)

98.9

(37.2)

19,431

(5.7)

1.14

(4.3)

90.3

(32.4)

17,339

(5.1)

35,858

(10.5)

36,771

(10.8)

47.9% -1.5%

Using 1.14 gpm (4.3 L/min) for the water flow rate in the August 8, 2013, the first test condition 

increases the water energy recovery percentage to 48%. For the other three August 8, 2013, test 

conditions, the backup flow meter reported 0.41 gpm (1.6 L/min), 0.55 gpm (2.1 L/min), and 

0.60 gpm (2.3 L/min), respectively.

Table 5. Mean Server Data for August 8, 2013, Test Conditions

Test Condition,

August 8, 2013

CPU-1 Temp,

°F

(°C)

CPU-2 Temp,

°F

(°C)

Server Fan

Speeds,

rpm

50% LINPACK, 

Setpoint 100°F

9:40 – 9:50:59

90.0

(32.2)

92.1

(33.4)

4043

50% LINPACK, 

Setpoint 120°F

12:50 – 13:00:59

109.5

(43.1)

111.2

(44.0) 4044

100% LINPACK, 

Setpoint 120°F

14:40 – 14:50:59

135.1

(57.3)

137.7

(58.7) 4039

100% LINPACK, 

Setpoint 140°F

16:40 – 16:50:59

133.5

(56.4)

136.3

(57.9) 4040

The mean server fan speeds show little variation from this point forward with the Asetek 

RackCDU in place, with mean fan speeds ranging from 4,039 rpm to 4,067 rpm over the 

different test conditions. Therefore, the 700 cfm airflow value used in the following tables was 

not directly measured but approximated due in part to setup time when only limited test windows

existed. There is one exception, and that is for the special case described in Section 3.3.1 that 

involved disabling server fans, as that involves a change in operating conditions.

Performance data were also collected at random times in the July–August 2013 timeframe while 

the rack was in normal system operation. The results appearing in Tables 6 and 7 were selected 

as the rack remained in a semi-steady state for at least an hour while data were collected.

For the July 19, 2013, tests:

• Mean inlet air temperature (Tin, air) remained steady at 68.5°F (20.3°C).

• Mean water facility supply temperature (Tin, water) remained steady at 62.3°F (16.8°C).

• Based on rack power data, this represents a more typical workload.
16

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

For the August 13, 2013, tests:

• Mean inlet air temperature (Tin, air) remained steady at 68.4°F (20.2°C).

• Mean water facility supply temperature (Tin, water) remained steady at 62.3°F (16.8°C).

• Based on rack power data, this represents an idle state.

Table 6. Mean Energy Flow Rates, Two States While Rack in Production (Hour Timeframe)

Normal Rack

Operation Data

Air Water Qtotal,

Btu/hr

(kW)

Rack 

Power,

Btu/hr

(kW)

Water

Energy

Recovery

Percent

Missing

Energy

Percent

Flow1,

cfm

(L/min)

Tout,

°F

(°C)

Qair,

Btu/hr

(kW)

Flow,

gpm

(L/min)

Tout,

°F

(°C)

Qwater,

Btu/hr

(kW)

July 19, 2013, 

Setpoint 100°F

13:00 – 13:59:59

700

(19,824)

99.8

(37.7)

19,713

(5.8)

1.08

(4.1)

93.6

(34.3)

17,008

(5.0)

36,721

(10.8)

38,148

(11.2)

44.6% 3.7%

August 13, 2013, 

Setpoint 100°F

11:00 – 11:59:59

700

(19,824)

93.51

(34.2)

15,838

(4.6)

1.05

(4.0)

79.9

(26.6)

9,203

(2.7)

25,041

(7.3)

24,226

(7.1)

38.0% -3.4%

Table 7. Mean Server Data, Two States While Rack in Production (Hour Timeframe)

Normal Rack

Operation Data

CPU-1 Temp,

°F

(°C)

CPU-2 Temp,

°F

(°C)

Server Fan

Speeds,

rpm

July 19, 2013, 

Setpoint 100°F

13:00 – 13:59:59

92.6

(33.6)

94.9

(35.0)

4049

August 13, 2013, 

Setpoint 100°F

11:00 – 11:59:59

68.2

(20.1)

69.4

(20.8) 4039

Additional graphs related to the liquid-cooled configuration appear in Appendix B. 

3.3 Second Retrofit Results

Opportunities were identified with Asetek and Aspen Systems to improve performance by 

reducing fan energy and thermal losses, so a second retrofit of the Skynet compute rack 

occurred. Section 3.3.1 reviews the aggressive changes attempted to reduce fan energy and 

results from a March 27, 2014, test that allowed one of the configurations to move into an 

operational test period. After it was reported the system was unstable at times for the Skynet user 

community, the rack configuration was reverted back on May 7, 2014, to running all the server 

fans in Energy Savings mode—marking the final configuration for the Skynet compute rack.

The RackCDU version 1.0 was also replaced with a new RackCDU version 1.1, although the 

server pump/cold plate units installed during the first phase of the project remained in place. 

Control parameters can be set for the RackCDU that include a minimum allowed facility flow 

rate through it; this value was set higher for the new RackCDU, and Section 3.3.2 contains the 

test results. Unfortunately, the backup reference flow meter stopped working during this time 

period (effectively reporting 0 gpm when facility water was flowing through), so results reported 

in the tables for March 27, 2014, and May 7, 2014, relied on the RackCDU internal flow meter.
17

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

3.3.1 Aggressive Reduction in Fan Energy Attempt

Finding opportunities to reduce server fan energy was very appealing to NREL. During the first 

phase of the project, it was observed that the mean server fan speeds showed little variation in 

the different test conditions after the RackCDU cooling solution was installed—so while Aspen 

System engineers were again on-site to monitor the Skynet system hardware, aggressive attempts

were made to lower the fan energy by disabling some of the server fans. Asetek came up with a 

jumper cable solution that could be installed that effectively allowed for two out of the four fans 

in a server to be disabled and for the speeds on the remaining two to be reduced. For the two fans 

that were disabled, it proved necessary to seal the face of the fan with (HVAC foil) tape to avoid 

airflow problems through the server (without it, the nodes in the server experienced thermal 

events). After successfully running a 100% LINPACK stress test on one of the modified servers, 

work began on converting the remaining 13 servers, starting with the top servers and running a 

LINPACK stress test after each server was modified. The RackCDU was also replaced during 

this planned system outage. With limited time to make these changes, a decision was made to 

only modify the top seven servers (with jumper cables to disable half of those fans) and leave the 

bottom seven servers in the original configuration (with all server fans running). This halfmodified configuration passed a short 100% LINPACK load condition test, and so the system

was turned back over to the Skynet user community for an operational test period.

Figure 9. Server fan configuration during aggressive second retrofit attempt.

Due to the replacement of the RackCDU, the thermocouples were not back in place for the 

March 27, 2014, test. In addition, there wasn’t time to put the capture hood in place, as airflow 

was definitely reduced in this configuration, so the air energy flow rate was not calculated in 

Table 8.

For the March 27, 2014, tests:

• Mean inlet air temperature (Tin, air) remained steady at 72.8°F (22.7°C).

• Mean water facility supply temperature (Tin, water) remained steady at 59.7°F (15.4°C).

Foil tape (on two fans)

Disabled fans
18

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Table 8. Mean Energy Flow Rates for March 27, 2014, Test Conditions

Rack Operation,

March 27, 2014

Special Case

Air Water Qtotal,

Btu/hr

(kW)

Rack 

Power,

Btu/hr

(kW)

Water

Energy

Recovery

Percent

Missing

Energy

Percent

Flow1,

cfm

(L/min)

Tout,

°F

(°C)

Qair,

Btu/hr

(kW)

Flow,

gpm

(L/min)

Tout,

°F

(°C)

Qwater,

Btu/hr

(kW)

100% LINPACK, 

Setpoint 100°F

17:49 – 17:59:59

Energy flow rate for air was 

not measured

1.7

(6.3)

91.9

(33.3)

26,727

(7.8)

?

Qair=?

42,857

(12.6)

62.4%

?

Qair=?

Table 9. Mean Server Data for March 27, 2014, Test Conditions

Rack Operation,

March 27, 2014

Special Case

CPU-1 Temp,

°F

(°C)

CPU-2 Temp,

°F

(°C)

Fan Speeds

Top Half of Rack

2 of 4 fans running,

rpm

Fan Speeds

Bottom Half of Rack

All 4 fans running,

rpm

100% LINPACK, 

Setpoint 100°F

17:49 – 17:59:59

99.2

(37.4)

103.1

(39.5)

2919 4076

During the operational test period, after the Skynet user community reported issues, the Ganglia 

CPU load averages were monitored, and nodes started showing loads of 9 though 20 at various 

times, mainly concentrated in the top half of the rack. A physical inspection was also conducted, 

and touching the metal connectors of Mellanox IB cables revealed a significant temperature 

difference when nodes were running loads of 15 though 20. There was also a noticeable airflow

difference in the upper half of the rack versus the lower half (which was expected).

3.3.2 Final Configuration with Updated RackCDU

The Skynet compute rack was returned to running all the server fans in the Energy Savings mode 

as the final system configuration state on May 7, 2014 (matching the project's first phase). A 

final set of tests were run, as the RackCDU version was updated from the first phase of the 

project.

For the May 7, 2014, tests:

• Mean inlet air temperature (Tin, air) remained steady at 72.4°F (22.5°C).

• Mean water facility supply temperature (Tin, water) remained steady at 61.6°F (16.4°C).

Table 10. Mean Energy Flow Rates for May 7, 2014

Rack Operation,

May 7, 2014

Air Water Qtotal,

Btu/hr

(kW)

Rack 

Power,

Btu/hr

(kW)

Water

Energy

Recovery

Percent

Missing

Energy

Percent

Flow1,

cfm

(L/min)

Tout,

°F

(°C)

Qair,

Btu/hr

(kW)

Flow,

gpm

(L/min)

Tout,

°F

(°C)

Qwater,

Btu/hr

(kW)

Idle State,

Setpoint 100°F

11:52 – 12:01:59

700

(19,824)

94.3

(34.6)

13,784

(4.04)

1.39

(5.26)

78.1

(25.6)

11,412

(3.34)

25,196

(7.4)

23,680

(6.9)

48.2% -6.4%

100% LINPACK,

Setpoint 100°F

12:12 – 12:21:59

700

(19,824)

99.56

(37.5)

17,048

(5.0)

1.98

(7.5)

89.3

(31.8)

27,572

(8.08)

44,619

(13.1)

42,720

(12.5)

64.5% -4.4%
19

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Table 11. Mean Server Data for May 7, 2014

Rack Operation,

May 7, 2014

CPU-1 Temp,

°F

(°C)

CPU-2 Temp,

°F

(°C)

Server Fan

Speeds,

rpm

Idle State,

Setpoint 100°F

11:52 – 12:01:59

68.4

(20.2)

69.9

(21.1)

4065

100% LINPACK,

Setpoint 100°F

12:12 – 12:21:59

93.0

(33.9)

96.8

(36.0)

4067

Additional material related to the second retrofit appears in Appendix C.

4 Conclusions

This project, conducted through a CRADA, allowed all the participants to better understand the 

process involved in retrofitting a production rack with the Asetek RackCDU cooling technology

and how that equipment responds in an energy-efficient data center like that found in ESIF. The 

project ended up having two distinct phases involving two versions of Asetek RackCDU, and it 

was during the second project phase that an aggressive attempt was made to actually disable 

some of the server fans.

4.1 Observations for the First Phase of the Project

The installation of the RackCDU version 1.0 was very straightforward, as a bolt-on rack 

extension attaches to the standard rack after connecting supply/return facility lines. The retrofit 

of the 14 servers took a three-person team 40 minutes each to convert and included running a test 

harness (so less than 10 hours total to retrofit the rack). At typical workloads, 44% to 48% 

energy recovery was observed with water returning around 95°F (35°C) that the ESIF ERW 

system can use as building heat source. But with all the server fans running in the Energy 

Savings mode, there were no noticeable fan energy savings—and thus no noticeable server 

power savings. Some of the test conditions led to water energy recovery percentages in the range 

of ~30%,which occurred when trying to drive the ERW temperature well above 95°F (35°C) into 

the 110°F (43°C) range for use as a heat source for buildings, domestic hot water, and other 

applications requiring heat. Aspen Systems commissioned the retrofit of the Skynet compute 

rack with the Asetek RackCDU cooling solution on June 10, 2013. 

Opportunities were identified with Asetek and Aspen Systems to improve performance by 

reducing fan energy and thermal losses. As a result, another retrofit of Skynet rack occurred, 

which also included a newer version of the RackCDU unit that retained the server pump/cold 

plate units.

4.2 Observations for the Second Phase of the Project

The RackCDU version 1.0 bolt-on rack extension was replaced with a new version 1.1; the 

installation was straightforward, as cooling liquid moves between the servers and rack extension 

utilizing hoses with dripless quick connects that are easy to disconnect/connect. An aggressive 

attempt was made to reduce fan energy, with an operational test period involving seven servers 

each operating with half of the server fans disabled starting in March 2014. This configuration 
20

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

passed a short LINPACK 100% load condition to move into an operational test period. However, 

during operational testing, some of the server air-cooled components caused the system to 

become unstable at times, and so the rack was converted back to running all the server fans in 

Energy Savings mode as the final system configuration state in May 2014. Ideally an OEM 

would include fan speed settings for when CPU and memory components are liquid cooled or 

allow custom settings. If the Skynet compute hardware had consisted of more than one similarly 

configured rack, perhaps more test opportunities would have been possible to explore different 

combinations to reduce server fan energy on a single rack—then after a successful operational 

test period, that solution could be rolled out to the remaining system racks.

At typical workloads in the final system configuration, 64% energy recovery was observed using 

a higher facility water flow rate, but lower return facility water temperature of 89°F (31.7°C). 

There are control parameters that can be set for the RackCDU that include a minimum allowed 

for facility flow through rack extension; the default value is set higher with the new version of 

RackCDU. The higher flow rates tested with the new RackCDU version correspond to a greater 

water energy recovery percentage; this matches observations made in a report by Lawrence 

Berkeley National Laboratory (LBNL) on a similar demonstration project. The study by LBNL 

included varying the supply water temperature and water flow rates at three different power 

levels (idle, 50%, and 100% load conditions) on servers that had Asetek pump/cold plate 

assemblies applied to both the CPU and memory components.

• “For a given supply water temperature, a higher flow rate corresponds to a greater 

fraction of heat captured. This behavior was expected because the water flow rate on the 

server side was constant, with the result that higher flow on the low-temperature side in 

turn dropped the temperature on the server side, thereby extracting a higher fraction of 

the heat from the direct-cooled components.” (Coles and Greenberg, 2014)

4.3 General Observations

When the 14 Skynet 2U servers were originally installed in NREL’s legacy data center, they had 

to be spread across two racks based on available air cooling options for that room. Using warmwater direct-to-chip liquid cooling solutions such as Asetek’s RackCDU to reject half the heat 

directly to liquid would allow density to be doubled in the case of the Skynet hardware. It might 

also allow for the server fan speed control mode to be changed to a more energy-efficient setting 

for additional savings.

After more than a year (going on 16 months) of the RackCDU system being in service on a 

production system, it has proven to be a very low-maintenance cooling system with high 

reliability—as no maintenance has been required during this time period and there have been no 

leaks. The RackCDU system was easy to integrate as a retrofit-in-place option.

5 Suggestions for Further Work

NREL has partnered with Asetek on a Department of Defense (DoD) Environmental Security 

Technology Certification Program (ESTCP) project to test the RackCDU in a DoD facility in the 

Southeastern United States. The results of the work at NREL have been incorporated in the 

ESTCP project.
21

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

References

ASHRAE (American Society of Heating, Refrigerating and Air-Conditioning Engineers). 2011 

Thermal Guidelines for Data Processing Environments – Expanded Data Center Classes and 

Usage Guidance. Atlanta, GA. 2011.

Coles, H.; Greenberg, S. (2014). Direct Liquid Cooling for Electronic Equipment. LBNL 

(Lawrence Berkeley National Laboratory). 51pp.; LBNL Report No. LBNL-6641E.

Sheppy, M.; Lobato, C.; Van Geet, O.; Pless, S.; Donovan, K.; Powers, C. (2011). Reducing 

Data Center Loads for a Large-Scale, Low-Energy Office Building: NREL's Research Support 

Facility. NREL (National Renewable Energy Laboratory). 32 pp.; NREL Report No. BK-7A40-

52785.
22

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Appendix A: Air-Cooled Data

Semi-steady-state time windows selected for May 16, 2013:

• 9:21 – 9:31:59 for 50% LINPACK test condition

• 9:47 – 9:57:59 for 100% LINPACK test condition.

600

650

700

750

800

850

900

950

1000

1050

1100

9:00 9:10 9:20 9:30 9:40 9:50 10:00

Air Flow (cfm)

Time

Air Flow for May 16, 2013

100

105

110

115

120

125

9:00 9:10 9:20 9:30 9:40 9:50 10:00

Tout (F)

Time

Air Tout for May 16, 2013
23

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

6000

7000

8000

9000

10000

11000

12000

13000

14000

15000

9:00 9:10 9:20 9:30 9:40 9:50 10:00

Power (kW)

Time

Rack Power for May 16, 2013

80

90

100

110

120

130

140

150

160

170

180

9:00 9:10 9:20 9:30 9:40 9:50 10:00

Temperature (F)

Time

CPU Temperatures for May 16, 2013

CPU-1

CPU-2
24

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Appendix B: Liquid-Cooled Data

Semi-steady-state time windows selected for August 8, 2013:

• 9:40 – 9:50:59 for 50% LINPACK test condition with desired facility water return 

setpoint at 100°F

• 12:50 – 13:00:59 for 50% LINPACK test condition with desired facility water return 

setpoint at 120°F

• 14:40 – 14:40:59 for 100% LINPACK test condition with desired facility water return 

setpoint at 120°F

• 16:40 – 16:50:59 for 100% LINPACK test condition with desired facility water return 

setpoint at 140°F.

600

650

700

750

800

850

900

950

1000

1050

1100

9:00 11:00 13:00 15:00 17:00

Air Flow (cfm)

Time

Air Flow for August 8, 2013
25

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

6000

7000

8000

9000

10000

11000

12000

13000

14000

15000

9:00 11:00 13:00 15:00 17:00

Power (kW)

Time

Rack Power for August 8, 2013

60

70

80

90

100

110

120

130

140

150

160

9:00 11:00 13:00 15:00 17:00

Temperature (F)

Time

CPU Temperatures for August 8, 2013

CPU-1

CPU-2
26

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Appendix C: Second Retrofit Attempt

Semi-steady-state time windows selected for May 7, 2014:

• 11:52 – 12:01:59 for Idle State condition with desired facility water return setpoint at 

100°F

• 12:12 – 12:21:59 for 100% LINPACK test condition with desired facility water return 

setpoint at 100°F.

70

75

80

85

90

95

100

105

110

115

120

11:40 11:50 12:00 12:10 12:20 12:30 12:40

Tout (F)

Time

Water Tout for May 7, 2014

6000

7000

8000

9000

10000

11000

12000

13000

14000

15000

11:40 11:50 12:00 12:10 12:20 12:30 12:40

Power (kW)

Time

Rack Power for May 7, 2014
27

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

60

70

80

90

100

110

120

130

140

150

160

11:40 11:50 12:00 12:10 12:20 12:30 12:40

Temperature (F)

Time

CPU Temperatures for May 7, 2014

CPU-1

CPU-2
28

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Appendix D: Testing – Lessons Learned

• Adjusting the RackCDU pulse-width modulation (PWM) setting that controls the facility 

water modulating pressure independent control valve in an attempt to quickly change test 

conditions prevented a semi-steady-state condition from being reached. This adjustment 

attempt was made only during the August 8, 2013, round of testing. Hunting behavior by 

the valve was observed where it would open and close rapidly to adjust the flow, versus 

steadily changing and leveling off to a semi-steady-state flow during test conditions.

• The array of thermocouples placement is important. The nine thermocouples were 

arranged in a 3x3 pattern spaced evenly within the profile of the servers (see Figure 10). 

However, the three thermocouples along the right side were not actually in a strong 

enough airstream due to the physical layout of the servers. A thin paper test revealed this; 

the observation was made while moving paper slowly from left to right across the back of 

the servers. Data revealed this behavior before and after the RackCDU retrofit as shown 

in Figures 11 and 12.

Figure 10. Temperature sensor arrangement (back of rack) and chassis back view. Images from 

Super Micro Computer, Inc.

a b c

d

e

f

g h i
29

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Figure 11. Thermocouple array, back of rack air temperatures for March 16, 2013.

Figure 12. Thermocouple array, back of rack air temperatures for August 8, 2013.
30

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Appendix E: Tables from Section 3 Grouped Together

Air-cooled configuration (Table 2)

Test Condition,

May 16, 2013

CPU-1 Temp,

°F

(°C)

CPU-2 Temp,

°F

(°C)

Server Fan

Speeds,

rpm

50% LINPACK

9:21 – 9:31:59

125.0

(51.7)

134.9

(57.2)

4501

100% LINPACK

9:47 – 9:57:59

150.4

(65.8)

156.8

(69.3)

5711

Liquid-cooled first phase configuration (Table 5)

Test Condition,

August 8, 2013

CPU-1 Temp,

°F

(°C)

CPU-2 Temp,

°F

(°C)

Server Fan

Speeds,

rpm

50% LINPACK, 

Setpoint 100°F

9:40 – 9:50:59

90.0

(32.2)

92.1

(33.4)

4043

50% LINPACK, 

Setpoint 120°F

12:50 – 13:00:59

109.5

(43.1)

111.2

(44.0) 4044

100% LINPACK, 

Setpoint 120°F

14:40 – 14:50:59

135.1

(57.3)

137.7

(58.7) 4039

100% LINPACK, 

Setpoint 140°F

16:40 – 16:50:59

133.5

(56.4)

136.3

(57.9) 4040

Liquid-cooled first phase configuration (Table 7)

Normal Rack

Operation Data 

CPU-1 Temp,

°F

(°C)

CPU-2 Temp,

°F

(°C)

Server Fan

Speeds,

rpm

July 19, 2013, 

Setpoint 100°F

13:00 – 13:59:59

92.6

(33.6)

94.9

(35.0)

4049

August 13, 2013, 

Setpoint 100°F

11:00 – 11:59:59

68.2

(20.1)

69.4

(20.8) 4039

Liquid-cooled second phase configuration (Table 11)

Rack Operation, 

May 7, 2014

CPU-1 Temp,

°F

(°C)

CPU-2 Temp,

°F

(°C)

Server Fan

Speeds,

rpm

Idle State, 

Setpoint 100°F

11:52 – 12:01:59

68.4

(20.2)

69.9

(21.1)

4065

100% LINPACK, 

Setpoint 100°F

12:12 – 12:21:59

93.0

(33.9)

96.8

(36.0)

4067
31

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Air-cooled configuration (Table 1)

Test Condition,

May 16, 2013

Air n/a Qtotal,

Btu/hr

(kW)

Rack 

Power,

Btu/hr

(kW)

n/a Missing

Energy

Percent

Flow,

cfm

(L/min)

Tout,

°F

(°C)

Qair,

Btu/hr

(kW)

50% LINPACK,

9:21 – 9:31:59

786

(22,260)

117.6

(47.6)

34,309

(10.1)

34,309

(10.1)

35,418

(10.4)

3.1%

100% LINPACK,

9:47 – 9:57:59

968

(27,414)

123.0

(50.6)

46,958

(13.8)

46,958

(13.8)

47,804

(14.0)

1.7%

Liquid-cooled first phase configuration (Table 3)

Test Condition,

August 8, 2013

Air Water Qtotal,

Btu/hr

(kW)

Rack 

Power,

Btu/hr

(kW)

Water

Energy

Recovery

Percent

Missing

Energy

Percent

Flow,

cfm

(L/min)

Tout,

°F

(°C)

Qair,

Btu/hr

(kW)

Flow,

gpm

(L/min)

Tout,

°F

(°C)

Qwater,

Btu/hr

(kW)

50% LINPACK,

Setpoint 100°F

9:40 – 9:50:59

694

(19,654)

98.9

(37.2)

19,431

(5.7)

1.08

(4.1)

90.3

(32.4)

16,427

(4.8)

35,858

(10.5)

36,237

(10.6)

45.3% 1.0%

50% LINPACK, 

Setpoint 120°F

12:50 – 13:00:59

694

(19,654)

102.7

(39.3)

21,749

(6.4)

0.53

(2.0)

105.8

(41.0)

12,145

(3.6)

33,894

(9.9)

36,408

(10.7)

33.4% 6.9%

100% LINPACK, 

Setpoint 120°F

14:40 – 14:50:59

703

(19,909)

107.4

(41.9)

25,200

(7.4)

0.61

(2.3)

118.8

(48.2)

17,949

(5.3)

43,150

(12.6)

47,770

(14.0)

37.6% 9.7%

100% LINPACK, 

Setpoint 140°F

16:40 – 16:50:59

703

(19,909)

107.1

(41.8)

24,960

(7.3)

0.64

(2.4)

118.1

(47.8)

18,707

(5.5)

43,667

(12.8)

48,009

(14.1)

39.0% 9.0%

Liquid-cooled first phase configuration (Table 6)

Normal Rack

Operation Data

Air Water Qtotal,

Btu/hr

(kW)

Rack 

Power,

Btu/hr

(kW)

Water

Energy

Recovery

Percent

Missing

Energy

Percent

Flow1,

cfm

(L/min)

Tout,

°F

(°C)

Qair,

Btu/hr

(kW)

Flow,

gpm

(L/min)

Tout,

°F

(°C)

Qwater,

Btu/hr

(kW)

July 19, 2013, 

Setpoint 100°F

13:00 – 13:59:59

700

(19,824)

99.8

(37.7)

19,713

(5.8)

1.08

(4.1)

93.6

(34.3)

17,008

(5.0)

36,721

(10.8)

38,148

(11.2)

44.6% 3.7%

August 13, 2013, 

Setpoint 100°F

11:00 – 11:59:59

700

(19,824)

93.51

(34.2)

15,838

(4.6)

1.05

(4.0)

79.9

(26.6)

9,203

(2.7)

25,041

(7.3)

24,226

(7.1)

38.0% -3.4%

Liquid-cooled second phase configuration (Table 10)

Rack Operation,

May 7, 2014

Air Water Qtotal,

Btu/hr

(kW)

Rack 

Power,

Btu/hr

(kW)

Water

Energy

Recovery

Percent

Missing

Energy

Percent

Flow1,

cfm

(L/min)

Tout,

°F

(°C)

Qair,

Btu/hr

(kW)

Flow,

gpm

(L/min)

Tout,

°F

(°C)

Qwater,

Btu/hr

(kW)

Idle State, 

Setpoint 100°F

11:52 – 12:01:59

700

(19,824)

94.3

(34.6)

13,784

(4.04)

1.39

(5.26)

78.1

(25.6)

11,412

(3.34)

25,196

(7.4)

23,680

(6.9)

48.2% -6.4%

100% LINPACK, 

Setpoint 100°F

12:12 – 12:21:59

700

(19,824)

99.56

(37.5)

17,048

(5.0)

1.98

(7.5)

89.3

(31.8)

27,572

(8.08)

44,619

(13.1)

42,720

(12.5)

64.5% -4.4%