Preparing for Post-Exascale Computing

Jeffrey S. Vetter
With many contributions from ACSR Section and Colleagues

The 30th Anniversary Symposium of the Center for Computational Sciences
University of Tsukuba
13 Oct 2022
ORNL is managed by UT-Battelle, LLC for the US Department of Energy

ORNL is managed by UT-Battelle
for the US Department of Energy

https://www.ornl.gov/section/advanced-computing-systems-research (https://j.mp/acsrs)
vetter@computer.org

Highlights• 15 years to go from Exascale ideation to deployment– Reports and predictions• Current status– Systems (Frontier)– Exascale Computing Project• Exascale: What did we get right, get wrong, overlook?
• Post Exascale?
– Heterogeneous systems enabled by Heterogeneous integration and Chiplets– Codesign becomes even more important• Abikso: Microelectronics codesign project16

Exascale Reports (and predictions) from 2007 to 2014

61

My first trip to Tsukuba!
• International Exascale

Software Project• Meeting 3:

Tsukuba, Japan
Oct. 18-20, 200962

• International Exascale Software Project63

DOE HPC Roadmap to Exascale Systems
FY 2012FY 2016

Titan

Summit

ORNL

ORNL

decommissioned

Cray/AMD/NVIDIA

MiraANLIBM BG/Q

FY 2018

IBM/NVIDIA

Theta

ANLCray/Intel KNL

To date, only
NVIDIA GPUs

FY 2022

FY 2023

Exascale
Systems

ORNL

HPE/AMDPolaris

ANL

HPE/AMD/NVIDIA

AuroraANLIntel/HPE

Cori
LBNL

Sequoia
LLNL

IBM BG/Q

Trinity
LANL/SNL

Cray/Intel Xeon/KNL

AMD, Intel and
NVIDIA GPUs!

LBNL

Cray/Intel Xeon/KNL

64

FY 2021

HPE/AMD/NVIDIA

SierraLLNL

IBM/NVIDIA

LANL/SNL
HPE/Intel

LLNL

HPE/AMD
Version 2.0

66

68

Exascale Computing Project has three technical areas to meetnational goals
Performant mission and science applications @ scale
Foster application
development

Diversearchitectures

HPC
leadership

Application Development (AD)SoftwareTechnology (ST)

Hardwareand Integration (HI)Develop and enhance the predictive
capability of applications critical tothe DOE

Produce expanded and verticallyintegrated software stack to achieve
full potential of exascale computing

Integrated delivery of ECP
products on targeted systems at
leading DOE computing facilities

25 applications ranging fromnational security, to energy, earthsystems, economic security,materials, and data
76

Easeof use

80+ unique software
products spanning
programming models
and run times, mathlibraries, data andvisualization
https://www.exascaleproject.org/

6 vendors supported
by PathForward
focused on memory,node, connectivity
advancements;
deployment to facilities

Application Development KPP-1 and KPP-2 Readiness Overview

ExaFel (Blocked ROCm) – The project team noticed during a recent work around to compiler bugs preventing compilation of Spinifel relies on an
upstream compiler built locally by the OLCF team. Integration into a formal ROCm release is still pending, so we have chosen to mark this as formally
blocked even though the team can currently make progress with the unofficial compiler build.
13013Extreme-scale Scientific Software Stack(E4S)• E4S: HPC software ecosystem – a curated software portfolio• A Spack-based distribution of software tested for interoperability

and portability to multiple architectures

https://spack.io
Spack lead: Todd Gamblin (LLNL)• Available from source, containers, cloud, binary caches• Leverages and enhances SDK interoperability thrust• Not a commercial product – an open resource for all• Growing functionality: May 2022: E4S 22.05 – 100+ full release products

Community policies

Commitment to SW quality

Curated collection

Quarterly releases

The end of dependency hell

Release 22.2 – February

Turnkey stack

https://e4s.io

A new user experience

131

DocPortal

Single portal to all
E4S product info

Portfolio testing

Especially leadershipplatforms

Build caches
10X build time
improvement

Post-ECP Strategy
Commercial E4S, SSO

https://e4s.io
E4S lead: Sameer Shende (U Oregon)
Also includes other products, e.g.,
AI: PyTorch, TensorFlow, Horovod
Co-Design: AMReX, Cabana, MFEM

ECP is Improving the LLVM Compiler Ecosystem
LLVM•Very popular opensource compiler
infrastructure•Permissive license•Modular, welldefined IR allows
use by a lot ofdifferent languages,
ML frameworks, etc.
•Backend
infrastructure
allowing the efficientcreation of backends
for new(heterogeneous)
hardware.
•A state-of-the-art
C++ frontend,
CUDA support,
scalable LTO,
sanitizers and otherdebuggingcapabilities, andmore.

+ SOLLVE•Enhancing theimplementation ofOpenMP in LLVM
•Unified memory•Prototype OMP
features for LLVM
•OMP Optimizations+ PROTEAS-TUNE•Core optimization
improvements toLLVM•OpenMP offload•OpenACC capability
for LLVM•Clacc•Flacc•OMP test suite•Autotuning for
OpenACC andOpenMP in LLVM

•Tracking OMP
implementation
quality•Integration with Tau
performance tools•Training•SYCL characterizing
and benchmarking+ FLANG+ HPCToolkit•Developing an opensource, production
Fortran frontend•Improvements toOpenMP profilinginterface OMPT•Upstream to LLVM
public release•OMPT specification
improvements•Support for OpenMPand OpenACC•Refine HPCT for
OMPT improvements•Recently approved
by LLVM+ NNSA•Enhancing LLVM tooptimize template
expansion for
FlexCSI, Kokkos,RAJA, etc.
•Flang testing andevaluation•Kitsune and Tapir•Initial implementationof serial F77
compiler for CPUs
under review•Leading LLVM-DOEfork•Training

Other ECP activities with LLVM emerging organically.
Active involvement with broad LLVM community: LLVM Dev, EuroLLVM
ECP personnel had 10+ presentations at the 2020 Dev Meeting
132Vendors•Increasing
dependence on
LLVM•Many vendors importand redistribute
LLVM•Contributions andcollaborations withmany vendors
through LLVM
•AMD•ARM•Cray•HPE•IBM•Intel•NVIDIASo how did we do?

Reflections from ECP Panel at ECP AHM in May 2022134

15 years is an eternity in computing - How did our predictions do?

135Hits

MissesOverlooked• System power came in at
O(20MW) not O(1GW)• Few major software rewrites /evolution• So far, FORTRAN -> C++ isthe main conversion• ECP included applications,software, and hardware• ~70 teams, ~1000
researchers• IESP• Concurrency (1B-wayparallelism)• Open-source software• Systems deployed 4 years
later than expected (of 2018)• Programming systems are
multiplying andimmature/incomplete• Hardware diversity• Demise of vendor interest inHPC• Fault tolerance• Productive programming
models (ala AI/ML): Python,Jupyter, Julia• Cost of ECP + NRE +Procurements approaches $3.6BUSD• AI/ML is not predicted (oreven mentioned)• Cloud deployment models• Green/sustainable computingPondering Post-ExascaleComputing• Thinking about the next 10 years

Important ArchitecturalTrends• Heterogeneous integration• Chiplets• Ecosystems and Standards– CXL, UCIe, BoW, …
• Open-source Tools and IP– RISC-V, OpenLane,

Silicon Compiler, etc• Open foundries• Codesign will be more

important than ever137[DARPA ERI Summit
2018]Reimagining Codesign• 2021 Workshop• Four priority research

directions– Drive Breakthrough Computing

Capabilities with Targeted Heterogeneityand Rapid Design– Software and Applications that Embrace
Radical Architecture Diversity– Engineered Security and Integrity fromTransistors to Applications– Design with Data-Rich Processes• We must make codesign agile, moreaccurate, and use real workloads

https://www.osti.gov/biblio/1822198-reimagining-codesign-advanced-scientific-computing-unlocking-transformational-opportunities-future-computing-systems-science

7 Sep 2022

Abisko138

Abisko: Microelectronics
CodesignAbisko Microelectronics Codesign Overview

Applications
Motifs, Composition

Algorithms

Software
ISA, IRArchitecture
Circuit scale up,
Interconnects, PDK

Devices andCircuits
Compact models

Materials
141

Codesign

API, Motifs

Abisko Microelectronics Codesign Overview
Collaborator

1. Develop better techniques for codesign from algorithms todevices and materials
2. Design Spiking Neural Network chiplet that can be integrated
with contemporary computer architectures
3. Explore new devices and materials for the SNN chiplet(neuron, synapse, plasticity, etc.)
4. Design language abstractions and runtime
support for SNN chiplet

Motivation••

1. Develop better techniques for codesign from algorithms todevices and materials
2. Design Spiking Neural Network chiplet that can be integrated
with contemporary computer architectures
3. Explore new devices and materials for the SNN chiplet(neuron, synapse, plasticity, etc.)
4. Design language abstractions and runtime
support for SNN chiplet

Transportation
CMS Sensors

Applications
Motifs, Composition

F1TenthCollaborator

CMS

Abisko Microelectronics Codesign Overview

Motivation••

1. Develop better techniques for codesign from algorithms todevices and materials
2. Design Spiking Neural Network chiplet that can be integratedwith contemporary computer architectures
3. Explore new devices and materials for the SNN chiplet(neuron, synapse, plasticity, etc.)
4. Design language abstractions and runtime
support for SNN chiplet

Algorithms•••

ML: SLAYER, Whetstone, EONS,eProp, STDP
Non-ML: Graph algorithms, CSP
Simulators: NEST, Brian2

Applications

Transportation
CMS Sensors

Algorithms
API, Motifs

Motifs, Composition

F1TenthCollaborator

CMS

Abisko Microelectronics Codesign Overview

Motivation••

1. Develop better techniques for codesign from algorithms todevices and materials
2. Design Spiking Neural Network chiplet that can be integrated
with contemporary computer architectures
3. Explore new devices and materials for the SNN chiplet(neuron, synapse, plasticity, etc.)
4. Design language abstractions and runtime
Softwaresupport for SNN chiplet•••Transportation
CMS Sensors

Algorithms•••

ML: SLAYER, Whetstone, EONS,eProp, STDP
Non-ML: Graph algorithms, CSP
Simulators: NEST, Brian2

DSL and API for neuromorphic co-processing
Built on LLVM and MLIR
Portable across Abisko chiplet, GPUs, etc.

Algorithms
API, Motifs

Software
ISA, IRApplications
Motifs, Composition

F1TenthCollaboratorCMS

Abisko Microelectronics Codesign Overview

Motivation••

1. Develop better techniques for codesign from algorithms todevices and materials
2. Design Spiking Neural Network chiplet that can be integrated
with contemporary computer architectures
3. Explore new devices and materials for the SNN chiplet(neuron, synapse, plasticity, etc.)
4. Design language abstractions and runtime
Softwaresupport for SNN chiplet
••
•

•
•

ML: SLAYER, Whetstone, EONS,eProp, STDP
Non-ML: Graph algorithms, CSP
Simulators: NEST, Brian2

DSL and API for neuromorphic co-processing
Built on LLVM and MLIRPortable across Abisko chiplet, GPUs, etc.

Algorithms

SoftwareArchitecture

Design neuromorphic chiplet
RISC-V neuromorphic extensions
Heterogeneous integration withcontemporary technologies

Motifs, Composition

API, Motifs

ISA, IR•••

Transportation
CMS Sensors

Algorithms•

Applications

F1TenthCollaborator

CMS

Abisko Microelectronics Codesign Overview

Architecture
Circuit scale up,
Interconnects, PDK

2.5D and 3D integration

Simulation/Emulation

Chiplet Architectures• Design an (analog) SNN chiplet that canbe easily integrated with contemporarytechnologies– Heterogeneous integration with mixedprocesses– Compatible with existing processes• Extensive advances in chiplets,packaging, and heterogeneous
integration recently[DARPA ERI Summit 2018]– Open Domain-Specific Architecture– UCIe, BoW, TSMC SoIC-CoW, Intel Foveros• Using open toolchain and architecture toexplore chiplet designs: RISC-V,OpenLane

7 Sep 2022

Abisko[IEEE HIR 2021]

150

Evaluation of 2.5D Chiplet for Neuromorphic Computing

refineimprove

Sandia VO2 ECRAM

conv 0ASIC Flow for Digital NN (baseline)

3bits

794 um• Investigate the performance of fully customized
ASIC design for ultra-fast NN inference– ORNL: HLS and RTL

conv0– Geogia Tech: ASIC Synthesis and PDmaxpooling• Model details:– Fixed NN architecture with quantized weights

conv1– Experimented with 2bit or 3bit of inputs (limited byFermiLab implementation)

2bits 526 um• Flow:dense0– Vitis HLS to generate RTL– Catapult logic synthesis– Customized backend layout tool (incl. tech
mapping, placement and routing)dense1• Achieved clock frequency of 1~ 2GHz in a 28nm
technology
1.
2.
3.
4.
7 Sep 2022

Layer Parameters
Bitwidth of features and weightsBus width of Interface
Fixed / Loadable Weights

Python
DNN Layer
Generator

Layer
XilinxCL

Vivado Toolchain
AbiskoRTL

Catapult ASIC
Synthesis

ASIC Synthesis
155

Motivation••

1. Develop better techniques for codesign from algorithms todevices and materials
2. Design Spiking Neural Network chiplet that can be integrated
with contemporary computer architectures
3. Explore new devices and materials for the SNN chiplet(neuron, synapse, plasticity, etc.)
4. Design language abstractions and runtime
Softwaresupport for SNN chiplet•••

Transportation
CMS Sensors

Algorithms•••

Applications

ML: SLAYER, Whetstone, EONS,eProp, STDP
Non-ML: Graph algorithms, CSP
Simulators: NEST, Brian2

Algorithms
API, Motifs

DSL and API for neuromorphic co-processing
Built on LLVM and MLIR
Portable across Abisko chiplet, GPUs, etc.

Software
ISA, IR

2.5D and 3D integration

Architecture

Devices and Circuits••••

Design neuromorphic chiplet
RISC-V neuromorphic extensions
Heterogeneous integration with
contemporary technologies

ion insertion (reversible doping) sets analog states
mRaman captures transition linear,
non-linear switching
Will extend to 36x36 x-bar arrayElectronic and other optical spectroscopies

Architecture

Devices andCircuits
Compact models

Simulation/Emulation

MesaFAB ReRAM

Circuit scale up,Interconnects, PDK

TaOx ReRAM•••

Motifs, Composition

F1TenthCollaborator

CMS

Abisko Microelectronics Codesign Overview

ECRAM

Devices and Circuits• Goals–Harness the interplay between mobile defects (ions and vacancies) and electronic properties to realize functional elements
for spiking and non-spiking analog neuromorphic networks–

Create and validate small network models; generate device and network data for co-design–

Understand and mitigate radiation induced degradation mechanisms at the device and circuit level

Prototypes

Circuits

Devices
inputs1) ECRAM

outputVTE2) ReRAM TiN

O anionsexchange

Ta (15 nm)

switchingchannel-2-TaOx (5-10 nm)
TiN- - - - - ++++ +++ ++ + +(+) charged
vacancies

synapses
inputsspiking output
neuronExperimental TaOx ReRAM Conductance Distributions
Developed TaOx
weight mapping andprogramming routine
for optimizing
inference accuracy

200ohm spacing between resistance targets
100ohm spread between Rmin, Rmax

VTE

Resulting conductance distribution

TiN
Ta (15 nm)
TaOx(5-10 nm)
TiN-- - - - - ++++ +++ ++ + +Motivation••

1. Develop better techniques for codesign from algorithms todevices and materials
2. Design Spiking Neural Network chiplet that can be integrated
with contemporary computer architectures
3. Explore new devices and materials for the SNN chiplet(neuron, synapse, plasticity, etc.)
4. Design language abstractions and runtime
Softwaresupport for SNN chiplet•••

Transportation
CMS Sensors

Algorithms•••

Applications
Motifs, Composition

Algorithms

ML: SLAYER, Whetstone, EONS,eProp, STDP
Non-ML: Graph algorithms, CSP
Simulators: NEST, Brian2API, Motifs

DSL and API for neuromorphic co-processing
Built on LLVM and MLIR
Portable across Abisko chiplet, GPUs, etc.

Software
ISA, IR

2.5D and 3D integration

Architecture

Devices and Circuits••••

Design neuromorphic chiplet
RISC-V neuromorphic extensions
Heterogeneous integration with
contemporary technologies

Architecture

ion insertion (reversible doping) sets analog states
mRaman captures transition linear,
non-linear switching
Will extend to 36x36 x-bar array
Electronic and other optical spectroscopies

MesaFAB ReRAM

Devices andCircuits

ECRAM
Computing Discovery Platform

Compact modelsMaterials•••

Non-equilibrium probes to few nm
Data-driven modelingOn-demand neuromorphism

Simulation/Emulation

Circuit scale up,
Interconnects, PDK

TaOx ReRAM•••

F1TenthCollaborator

CMS

Abisko Microelectronics Codesign Overview

Domain wall memristor

Materials

Computational data mining

CNMS scanning probe
microscopy and chemical imaging

Conclusions

Recap• Exascale is here!
• Our predictions were reasonably accurate,

but we completely missed some– AI/ML• Visit us (post COVID )– We host interns and othervisitors year round• Faculty, grad, undergrad,high school, industry– Programming systems remain major challenge• Post-exascale– Heterogeneous integration and Chiplet
architectures are vastly diversifying thearchitectural landscape– Post exascale will be accelerated by recent
major semiconductor investments– Visit https://jobs.ornl.gov• Contact mevetter@ornl.gov• Abisko microelectronics codesign project• Experimental Computing• Start building your own chiplets today!

– Lots of emerging archs

developing a chiplet for analog SNN228• Jobs at ORNL

Lab

– https://excl.ornl.gov