Update on Testbeds at ADAC Partners
ORNL Experimental Computing Laboratory
Jeffrey S. Vetter
With many, many contributions from workshop participants, FTG Group, ExCL team, and colleagues

ADAC8Tokyo
30 Oct 2019

ORNL is managed by UT-Battelle, LLC for the US Department of Energy

Time for a short poll…

27

History

Q: Think back 10 years. How
many of you would havepredicted that many of our
top HPC systems would be
GPU-based architectures?

29YesNo
Revisionists Future
General purpose multicore CPU

Q: Think forward 10 years.
How many of you predictthat most of our top HPC
systems will have thefollowing architecturalfeatures?

GPUFPGA/Reconfigurable processor
Neuromorphic processor
Deep learning processor
Quantum processor
RISC-V processor
Some new unknown processor
All/some of the above in one SoC31

ADAC Emerging Technologies

ADAC Emerging Technologies Charter• Goal: create collaborative testbed environments where emerging technologies can be investigated to inform futurearchitectures and software and applications development• Motivation– Need very early access to technologies in this age of Extreme Heterogeneity– Investigating testbeds is different than using HPC production systems•

Usage models– software development– exclusive access benchmarking•

Privileges– Constantly (re)install software environment from hardware up including OS– Reconfigure firmware– Configure hardware•

Immature software and hardware• Share– Lessons– Software (e.g., containers, drivers, fixes)– Operations– Hardware
35– Security

https://doi.org/10.2172/1473756

Progression of Experimental Computing
Technologies
TRL 4-6 Emerging• Examples: FPGAs in HPC,
TrueNorth, SpiNNaker, DWave, Emu, many SoCbased systems, TPU, Gen-ZNoCs, near-memorycomputing

TRL 1-3 Basic Concepts• Examples: carbonnanotube computing,memristor-basedneuromorphic computing,chip-level silicon
photonics, universal
quantum computing

TRL 7-9 Operational• Examples: Titan, Cori, Mira,
Summit, BlueWaters,
Keeneland, Stampede,
Tsubame2.5

Production
Experimental Prototype
Limited Access Testbed“Bench” System
Assembly language, or lessFew, if any, development tools

Language support and compilers.

OS-R

ManualSpecialized programming environments and OSs

Commodity OS & runtime systems

Scale

Small collections of devices

Single to hundreds of engineered processing elements>10,000 processing elements

Performance

Analytical projections basedon device empiricalevaluation.

Analytical projections or simulation based on component
or pilot system empirical evaluation.

Empirical evaluation of prototype and finalsystems.

Apps

Small encoded kernels

Architecture-aware algorithms; Mini-apps; Small
applications

Numerical libraries; Full scale applications

ExampleGPUs invented in 1999

OpenGL in 2001; CUDA in 2007; OpenCL in 2008; OpenACC

GPUs are a fully supported compute

Programming44

CS & Math ResearchLevels of Privileged Access
Application-level benchmarking and software development

Modify installed software and tools

More Users

Longer Experiments

More ExCL Resources
46

Modify installed drivers; low-level power
measurements
Bare metal:
Modify/replace OS, kernel
level experimentation
Hardware andfirmwaremods

ORNL Experimental Computing Laboratory (ExCL)

Project and User management

Community•Accounts•Projects and Proposals•Help•Workshops•Online discussions forums and issues•Consolidated•News

GPU: PASCAL, VOLTA, DGX, Vega
FPGA: Arria, Stratix
NVM: Intel Optane, NVMe
Deep memory: HBM2

Shared Login and Gateway Nodes

Authentication and Authorization•Gateway nodes•Data transfer nodes•Consistent and secure access to private
network compartments•Secure operations•Partition access to specific compartments•System and account lifecycles•Experience with management of export
controlled and proprietary systems

SoC: Zynq, Snapdragon, Xavier
Data intensive: Emu
Cloud: OpenStack Cluster

Shared Filesystems and Databases

Monitoring and control systems•Secure access to filesystems across pillars•Manage access to shared resources•Manage privileged access levels•Lights out operation

Containers: Docker
Cryogenic devices: JJ memory cell
Neuromorphic: TrueNorth, Lohi

Source Code and Data sets

Web

•Source Code repos•Performance databases for applications
and architectures•Educational and reference materials•Outreach•Both Open and Controlled access

Quantum: Rigetti, IBM, D-wave
Deep Learning: Cerebras, SambaNova
This year’s hot item

47

Pillars refreshed annually

Per pillar expert collaboration

ExCL Common Infrastructure

ExCL Technology Pillars

ExCL 2.0 (ORNL)
Exclusive Access Cluster
Management
ServerWeb portal for bare-metaland gateway VMs
management

Login
Node

Dedicated login node.
Creates the idea of an
integrated system

48

Bare-Metal Node Type A
Bare-Metal Node Type A
Bare-Metal Node Type A
Bare-Metal Node Type B
Bare-Metal Node Type C
Bare-Metal Node Type D
Gateway
Gateway
Gateway

IP/key based
restrictedaccessHardware wherevirtualization is notpossible. Only accessible
from gateways

Special HW 1Special HW 2

Virtual login nodes on top of VMs (not baremetal). Only to ensure that 1 user is goingto access other HW from here at a time.
Gateway machines can have associatedmetadata to make them unique.

Shared Node Type A
Shared Node Type B

Compute Nodes

Exclusive access tomachines in this cluster.
These nodes are onlyavailable once a VM has
been launched from theweb portal

Hardware not located at
ORNL. ie. quantum system

Special HW External
Same type of HWavailable in bare-metal,
but shared. No VM
needed. Direct accessfrom login node. Multiple
concurrent users

49

58

https://excl.ornl.gov/

Apache Pass Optane-based Memory System
Experimental Computing Lab (ExCL) managed by the ORNL Future Technologies Group• Intel OPTANE Memory– 1.5 TB of OPTANE Memory (Persistent)• 12 * 126 GiB NV DIMMS (2666 MHz)– 384 GiB of DRAM (Volatile)• 12 * 32 GiB DRAM (2933 MHz)

• Accessed as filesystem or memory access mode
/ configurable at boot time––––

Most recent Linux Kernel deployed (5.2.0)
Intel PMM drivers and PMM tools deployed
Newer kernels built and deployed on request
Kernel-matched perf command to read memory
performance and health instrumentation• Supermicro X11DPU-Z+ motherboard––––59Intel Gold 6248
2 CPUs * 20 cores/socket * 2 threads/core
Mellanox ConnectX-4 networking
12 NVMe SSDs

https://software.intel.com/en-us/articles/configure-manageand-profile-intel-optane-dc-persistent-memory-modules

For more information or to apply for an account, visit https://excl.ornl.gov/

Mar 2019

https://excl.ornl.gov/

Intel Stratix 10 FPGA

Experimental Computing Lab (ExCL) managed by the ORNL Future Technologies Group• Intel Stratix 10 FPGA and four banks ofDDR4 external memory– Board configuration: Nallatech 520 NetworkAcceleration Card

• Up to 10 TFLOPS of peak single precisionperformance• 25MBytes of L1 cache @ up to 94 TBytes/speak bandwidth• 2X Core performance gains over Arria® 10• Quartus and OpenCL softwarev18.1) for using FPGA(Intel SDK• Provide researcher access to advanced
FPGA/SOC environment60

For more information or to apply for an account, visit https://excl.ornl.gov/Mar 2019https://excl.ornl.gov/

NVIDIA Jetson AGX Xavier SoC

Experimental Computing Lab (ExCL) managed by the ORNL Future Technologies Group• NVIDIA Jetson AGX Xavier: High-performancesystem on a chip for autonomous machines• Heterogeneous SoC contains:– Eight-core 64-bit ARMv8.2 CPU cluster (Carmel)– 1.4 CUDA TFLOPS (FP32) GPU with additional
inference optimizations (Volta)– 11.4 DL TOPS (INT8) Deep learning accelerator(NVDLA)– 1.7 CV TOPS (INT8) 7-slot VLIW dual-processor
Vision accelerator (PVA)– A set of multimedia accelerators (stereo, LDC,optical flow)• Provides researchers access to advanced highperformance SOC environment

61

For more information or to apply for an account, visit https://excl.ornl.gov/

Mar 2019

https://excl.ornl.gov/

Qualcomm 855 SoC (SM8510P)

Experimental Computing Lab (ExCL) managed by the ORNL Future Technologies Group
Qualcomm Development Board connected to (mcmurdo) HPZ820
7nm TSMC

Adreno 640

5GHexagon 690
Spectra 360
Kyro 485© Qualcomm Inc.
© Qualcomm Inc.

Connectivity (5G)

Kyro 485 (8-ARM Prime+BigLittle Cores)
PrimeCore• Snapdragon X24 LTE (855 built-in) modem LTE Category 20• Snapdragon X50 5G (external) modem (for 5G devices)• Qualcomm Wi-Fi 6-ready mobile platform: (802.11ax-ready,802.11ac Wave 2, 802.11ay, 802.11ad)

Hexagon 690 (DSP + AI)• Quad threaded Scalar Core

• DSP + 4 Hexagon Vector Xccelerators• New Tensor Xccelerator for AI62

• Qualcomm 60 GHz Wi-Fi mobile platform: (802.11ay,802.11ad)• Bluetooth Version: 5.0• Bluetooth Speed: 2 Mbps

• Apps: AI, Voice Assistance, AV codecs• High accuracy location with dual-frequency GNSS.

Adreno 640• Vulkan, OpenCL, OpenGL ES 3.1

Spectra 360 ISP• New dedicated Image Signal Processor (ISP)• Apps: HDR10+, HEVC, Dolby, etc• Dual 14-bit CV-ISPs; 48MP @ 30fps single camera• Enables 8k-360o VR video playback• Hardware CV for object detection, tracking, streo depth process• 20% faster compared to Adreno 630• 6DoF XR Body tracking, H265, 4K60 HDR video capture, etc.

• Connected Qualcomm board to HPZ820 through USB• Development Environment: Android SDK/NDK• Login to mcmurdo machine$ ssh –Y mcmurdo• Setup Android platform tools and development environment$ source /home/nqx/setup_android.source• Run Hello-world on ARM cores$ git clone https://code.ornl.gov/nqx/helloworld-android$ make compile push run• Run OpenCL example on GPU$ git clone https://code.ornl.gov/nqx/opencl-img-processing• Run Sobel edge detection$ make compile push run fetch• Login to Qualcomm development board shell$ adb shell$ cd /data/local/tmp

For more information or to apply for an account, visit https://excl.ornl.gov/

Created by Narasinga Rao Miniskar, Steve Moulton

67

DevOps and CI/CD Tools Available
CI/CD Builder Envs

68

Containers / Orchestration• GitLab Auto DevOps• FTG Kuber in AWS• AWS Code Pipeline• Docker in ExCl & CADES• Circle CI• OpenShift support in process

Work in Progress• Procure and deploy

relevant AI testbeds• Finalize containerstrategy• Add users

69

https://basicmi.github.io/AI-Chip/