NREL is a national laboratory of the U.S. Department of Energy

Office of Energy Efficiency & Renewable Energy

Operated by the Alliance for Sustainable Energy, LLC

This report is available at no cost from the National Renewable Energy

Laboratory (NREL) at www.nrel.gov/publications.

Contract No. DE-AC36-08GO28308

CRADA Report

NREL/TP-2C00-73293

February 2019

Research and Testing of Power Use 

of HPC Workload of Participant’s HPC 

System

Cooperative Research and Development 

Final Report

CRADA Number: CRD-16-626

NREL Technical Contact: Kevin Regimbal
NREL is a national laboratory of the U.S. Department of Energy

Office of Energy Efficiency & Renewable Energy

Operated by the Alliance for Sustainable Energy, LLC

This report is available at no cost from the National Renewable Energy

Laboratory (NREL) at www.nrel.gov/publications.

Contract No. DE-AC36-08GO28308

National Renewable Energy Laboratory

15013 Denver West Parkway

Golden, CO 80401

303-275-3000 • www.nrel.gov

CRADA Report

NREL/TP-2C00-73293

February 2019

Research and Testing of Power Use 

of HPC Workload of Participant’s HPC 

System

Cooperative Research and Development 

Final Report

CRADA Number: CRD-16-626

NREL Technical Contact: Kevin Regimbal

Suggested Citation

Regimbal, Kevin. 2019. Research and Testing of Power Use of HPC Workload of Participant’s 

HPC System: Cooperative Research and Development Final Report, CRADA Number 

CRD-16-626. Golden, CO: National Renewable Energy Laboratory. NREL/TP-5000-73293.

https://www.nrel.gov/docs/fy19osti/73293.pdf.
NOTICE

This work was authored by the National Renewable Energy Laboratory, operated by Alliance for Sustainable 

Energy, LLC, for the U.S. Department of Energy (DOE) under Contract No. DE-AC36-08GO28308. Funding 

provided by the U.S. Department of Energy Office of Science (SC), Advanced Scientific Computing Research 

(ASCR), Scientific Discovery through Advanced Computing (SciDAC). The views expressed herein do not 

necessarily represent the views of the DOE or the U.S. Government.

This report is available at no cost from the National Renewable 

Energy Laboratory (NREL) at www.nrel.gov/publications.

U.S. Department of Energy (DOE) reports produced after 1991 

and a growing number of pre-1991 documents are available 

free via www.OSTI.gov.

Cover Photos by Dennis Schroeder: (clockwise, left to right) NREL 51934, NREL 45897, NREL 42160, NREL 45891, NREL 48097, 

NREL 46526.

NREL prints on paper that contains recycled content.
1

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

Cooperative Research and Development Final Report

Report Date: 10/18/18

In accordance with requirements set forth in the terms of the CRADA agreement, this document 

is the final CRADA report, including a list of subject inventions, to be forwarded to the DOE 

Office of Science and Technical Information as part of the commitment to the public to 

demonstrate results of federally funded research.

Parties to the Agreement: Colorado School of Mines

CRADA number: CRD-16-626

CRADA Title: Research and Testing of Power Use of HPC Workload of Participant’s HPC System

Joint Work Statement Funding Table showing DOE commitment:

Estimated Costs NREL Shared Resources

a/k/a Government In-Kind

Year 1

Year 2 or Modification #

Year 3 or Modification #

Year 4 or Modification #

$ 59,616.00

$ 64,799.00

$ 59,710.00

TOTALS $ 184,125.00

Abstract of CRADA Work:

This CRADA (1) facilitates relocation of the Board of Trustees of the Colorado School of Mines, 

for and on behalf of the Colorado School of Mines, (MINES or Participant) International 

Business Machines Corporation (IBM) supercomputing system into the National Renewable 

Energy Laboratory’s (NREL or Contractor) Energy Systems Integration Facility (ESIF). This 

CRADA (1) facilitates testing High Performance Computing (HPC) application platform 

portability, scalability, electrical energy use, and performance for applications critical to each 

organization; (2) facilitates use of the existing facility electrical and thermal data collection 

systems at Contractor ESIF in support of characterizing HPC workload power use 

characterization; and (3) facilitates addition of application power use data on the Participant IBM 

supercomputing system into the Contractor ESIF Time Series data system to use/re-use existing 

methodology to profile application power use.
2

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications

Summary of Research Results:

Task 1: Relocation of Participant’s IBM supercomputing system to Contractor ESIF HPC Data 

Center.

Contractor Responsibilities:

1. Designate local representatives who shall, on occasion as mutually agreed upon, perform 

such tasks as power cycling of equipment.

2. In case of an emergency, perform an Emergency Power Outage (EPO) and/or undertake 

any other action(s) necessary.

3. Maintain the supporting infrastructure needed to operate the computer as described in the 

CRADA, including providing the floor space, cooling, and other environmental systems 

necessary for the system while it is located at the ESIF.

4. Provide the same level of physical security, access security, monitoring, and fire 

protection for the system that is provided for comparable contractor resources housed in 

the ESIF.

5. Ensure that the system has access and connections to power and back-up power sources 

consistent with that which is provided for comparable contractor resources housed in the 

ESIF.

6. Provide reasonable physical access to the system for designated participant and its 

subcontractors’ personnel, including the right to enter the ESIF with reasonable notice 

and during normal business hours, and use it for the Purpose and for the Term of the

CRADA. Contractor agrees to provide participants’ designated personnel with badges or 

other entry devices, as necessary and with reasonable notice.

7. Provide non-proprietary and/or non-confidential information to participant as requested 

in order for participant to determine that the ESIF infrastructure is adequate to support the 

system.

8. Provide fiber optic connectivity in the contractor ESIF to the participant provided 

network device.

9. Provide a pilot project account on Peregrine to a designated participant representative to 

support the collaborative research described in the Purpose section.

This activity was completed successfully. The IBM supercomputing system was installed in the 

NREL ESIF HPC data center.

Task 2: Initiate Data Collection

Contractor Responsibilities:

1.Provision data collection from the facilities meters into the ESIF Time Series database.

2.Process CSV files provided by participant to load data into the ESIF Time Series database.
3

This report is available at no cost from the National Renewable Energy Laboratory (NREL) at www.nrel.gov/publications.

NREL verified that facility data collection was working as expected.

CSV files were not provided by participant to NREL. Without data from the IBM 

supercomputing system, Task 3 and Task 4 could not be completed.

Task 3: Data Analysis

Contractor Responsibilities:

1. Apply application power analysis methodology to data from participant’s IBM 

supercomputing system

2. Host collaboration session to look at data to identify important applications and trends

Data analysis task was not completed. CSV files as described in the JWS were not provided to 

NREL.

Task 4: Reporting

Contractor Responsibilities:

1. Draft report which summarizes methodology, findings, and conclusions.

Reporting task was not completed. CSV files as described in the JWS were not provided to 

NREL.

The hardware was removed from the ESIF HPC datacenter in January, 2018. 

Subject Inventions Listing:

None

ROI #:

None

Responsible Technical Contact at Alliance/NREL:

Kevin Regimbal

Name and Email Address of POC at Company:

Timothy Kaiser, tkaiser@mines.edu

This document contains NO confidential, protectable, or proprietary information.