IXPUG Workshop HPC Asia 2023

Porting Simulation,
Data-Intensive, and AI
Applications to the
Aurora Exascale
SystemTIM WILLIAMS

VENKAT VISHWANATHSCOTT PARKER

Deputy Division Director
Computational Science Division (CPS)
Argonne National Laboratory

Data Science Team Lead
Argonne Leadership Computing Facility (ALCF)
Argonne National Laboratory

Performance Engineering Team Lead
Argonne Leadership Computing Facility (ALCF)
Argonne National Laboratory

27 February 2023

Aurora – Argonne’s Exascale Supercomputer
GPU

GPU

GPU

GPU

GPU

CPU

CPU

HBM

HBM

DRAM

DRAM

Slingshot

SlingshotGPU

Aurora node
§ 2X Intel® Xeon® CPU Max Series§ 2 exaFLOPS double precision§ 6X Intel® Data Center GPU Max Series§ > 10,000 nodes

Aurora I/O System§ HPE Slingshot 11 network, dragonfly§ Distributed Asynchronous Object Storage (DAOS)§ > 10 PB memory§ ≥ 230 PB, ≥ 25 TB/s
2

Status of Aurora§ Aurora is built with exception of compute blades§ Installation of compute blades well underway§ Targeting early-user system access in Q3

3

Exascale Science
Simulation

§
§
§
§
§

Data

Learning

Exascale CFD simulation data trains NN model for rapid design studies
Figure 1: Data flow and summary of the FRNN algorithm
Exascale cosmology simulation data, sky survey data, and ML inform observations and test theory
Missing a real disruption (false negative) can be costly because of the damaging effects of a disruption, while trig
Exascale training from tokamak experimental data for inference at experimentpositive alarm wastes valuable experimental time and resources. Setting the threshold allows a tradeoff between
Exascale inference from DNN to construct connectome from massive EM brain tissue dataset
4
Exascale search spaces: PV materials, chemical kinetics, cancer drug combinations, detector collision events

Exascale Applications/Software Readiness§ ALCF Aurora Early Science Program (ESP)§ DOE Exascale Computing Project (ECP)§ Managers: Tim Williams, Venkat Vishwanath§ 3 technical areas: Application Development,
Software Technology, Hardware and Integration– AD: 21 applications projects preparing
codes for exascale– ST: 66 unique software products– HI: Applications Integration: deploy apps on
specific exascale systems (Aurora,
Frontier)§ 9 Simulation,10 Data and Learning projects§ Every project will run a proposed sciencecampaign on Aurora§ Training: Workshops, Hackathons, Dungeon
Sessions, webinars
a proof-of-concept, RMG-Cat successfully discovered the major kinetic pathways forCH4 oxidation on nickel; in less than 5 minutes on a single core, RMG-Cat was able tofind all the same reactions as a microkinetic mechanism developed over several years bya team of experts. RMG-cat can use a single node effectively. An entire RMG-Cat run
can take between a few seconds to a few hours on a personal laptop, but for the purposesof this application we only need to know which thermodynamic and kinetic parameters
are absent or poorly estimated. This node-bound preprocessing step only takes a few
seconds to evaluate. Once these parameters have been calculated through the workflow
RMG-Cat can be run entirely as a post-processing step, and if necessary this process canbe iterated upon until all relevant species are calculated. For each species, and all related
reactions between those species that RMG-cat does not have information for, we will
launch an instance of our search suite (KinBot, GAlgo, LRT) in parallel.

§ Argonne postdoc and staff support (Catalysts)
Once the area to search is defined KinBot and GAlgo will use learning techniques such as
genetic algorithms to efficiently explore the PES. These efficient searches will require
hundreds of thousands (106) of individual energy evaluations and will simultaneously
learn a low rank tensor (LRT) approximation of the PES that will be used to expedite thesearch and to calculate the kinetic and thermochemical parameters.

Dark Sky Mining

Salman Habib

but not for the typical BSP-based supercomputing application. This model is very suited for the
A21 architecture (details cannot be discussed here due to RSNDA restrictions).
As a specific example from this project we consider the construction of synthetic galaxy
catalogs; these have become the centerpiece of theindustry standard practice for validating cosmologParticle Catalog from N-body Simulation
ical inference pipelines. While the use-cases ofsuch galaxy catalogs are highly varied, the princiHalo Finder
Core Catalog
pal idea behind their use is straightforward: the existence of a realistic mock universe generated from
Halo Catalog
Merger Tree
Generator
a known underlying model allows scientists to valMerger Trees
idate the analysis pipelines they intend to run on
Position Finder
Galacticus
the actual astronomical dataset.
Figure 1 shows an end-to-end diagram of
Galaxy Positions
Galaxies
Light-cone
the workflow for producing a synthetic sky from
Generator
Light-cone
raw simulation outputs, which requires an entire
Generator
Particleecosystem of scientific codes rather than a single
Light-cones
Galaxy Light-cone Positions,application. Beginning from (Level 1) direct simuVelocities, Redshifts
FOV Cut-outs
lation outputs, this ecosystem:
ProtoDC2 Catalog

Each of these searches will be carried out in the following manner, as shown in figure 1.
At each step, GAlgo proposes a configuration to evaluate an objective function thatmeasures the proximity of that particular configuration to a critical point of interest (e.g.
saddle point or a local minimum). The configuration is represented by RxN matrix, where
R is the number of configurations, and N is the dimensionality (e.g. for normal
coordinates N=3a-6 where a is the number of nuclei), while the objective function
requires KinBot to evaluate the PES from a computational chemistry application such as
NWChem, as well as gradients with respect to each dimension, leading to a matrix of size
Rx(N+1). GAlgo then uses the objective function value to propose a new configuration as
it proceeds with the search of saddle points. KinBot generates input and parses output
from simulations to feed these energies and gradients into LRT. LRT will need theaggregated number of PES evaluations input/output pairs, as MxN and Mx1 matrices,
respectively, where M is the total, aggregated number of PES evaluations. The
constructed LRT approximation (stored as a coefficient tensor) will be invoked instead of
the PES evaluation if GAlgo's imposed accuracy tolerance is met. Our current estimates
for the above dimensionalities are: N~100, M~106, and R~104.
RMG-Cat encounters
unknown chemistry

SubmitQuantum
Chemistry
Job to Queue

KinBot initiates
phase space search
for minima andtransition states

GAlgo inquires thesurface evaluation

High Accuracy
Single Point jobs

PES
or LRT

Properties stored
in database

5x5 sq. deg
0<z<1

Mass-sheet
Generator
Ray Tracing
Kappa Maps

Shear Interpolator
Shears at galaxy
positions

Properties include:
galaxy_id, ra, dec,
M*, SDSS ugriz,
shears,magnification,
host halo properties

2. Creates Level 3 mock galaxy catalogs by Figure 1. Workflow for synthetic galaxy catalog
training and applying nonlinear mappings construction showing science products (courtesy
from the halos to synthetic galaxies that re- LSST DESC Cosmological Simulations working
side in the halos.
group).
It is important to note that except for the open-source semi-analytic galaxy formation code Galacticus [16] (which for many tasks is already being replaced by an Argonne-based methodology), all
the functional blocks in Figure 1 exist, were written by the Argonne team, and are already running
as a parallel workflow application on Cetus, Cooley, and the Phoenix cluster (the last-named isa CELS/HEP data-intensive computing pathfinder project). Indeed, this science pipeline has now
reached production-level maturity: our group is currently deploying it to generate the flagship synthetic catalogs for LSST DESC’s Data Challenge 2. At the time of this writing, a growing list of
more than twenty distinct analyses have been proposed by LSST DESC members who will use
these catalogs as a foundational component of the methodology in their proposed publications.
For our Aurora application, we propose to scale up our science pipeline to generate large suitesof high-accuracy synthetic skies, making the data products publicly available for analysis by thewider cosmological community. The production phase of this proposal will be preceded by a training phase, in which we fine-tune our hierarchical model with targeted application of machine learning algorithms. As described below in Section 2.5.4, we will use generative deep learning models
trained on high-resolution simulations to produce synthetic halo- and star-formation histories. We
will accelerate evaluation of our forward modeled summary statistics with Gaussian Process modeling, applying Variational Autoencoder techniques for dimensional reduction. Because our goal
6

K. E. Jansen

Figure 1: Application workflow

FOV Galaxy Locations

1. Creates Level 2 science datasets such as catalogs of dark matter halos, as well as timeseries data such as trees of merger histories
of each halo;

Data/Learning for Exascale CFD

LRTapproximation of
the surface

5

Figure 2: Isosurface of instantaneous Q criterion colored by speed over a vertical tail at Re = 3.5 105 ,with a rudder deflection angle of 30 degrees and 12 unsteady jets active. This DES simulationshows our method’s ability to refine the grid to capture the unsteady structures resulting from theseparation near the rudder and from the interaction between the unsteady jets and the crossflow.

2e. Application Summary(a) CFD - First adapted mesh.

(b) CFD - Second adapted mesh.

(c) Experiments.

Figure 3: Phase-averaged isosurface of velocity (color) and vorticity (grey) revealing coherent structures in the wake of a synthetic jet located at the junction between the stabilizer and the deflected
rudder of a vertical tail. Comparison between CFD predictions on two successive adapted meshes
and experimental results (c).

16

The proposed simulations embody a complex
workflow, which integrates several codes andalgorithms, as illustrated in Figure 1. Data will be
generated by the electronic structure layer, comprising
the Quantum Espresso9-10 code for generating low-cost,
low-fidelity DFT data and the BerkeleyGW code for
Figure 1:high-cost,
Data flow and summaryof the FRNN algorithm MBPT data. Data
generating
high-fidelity
Missing a real disruption(false negative) cansimulations
be costly because of the damaging
of a disruption,from quantum
mechanicalwill effects
flow
into while triggering a false
positive alarm wastes valuable experimental time and resources. Setting the threshold allows a tradeoff between these two
two layers of machine learning, a discovery layer and adecision layer. The discovery layer includes feature
selection, using the SISSO code, to identify low-costfeatures that correlate strongly with high-cost excited Figure 1: Workflow of the proposed simulations
state properties, as well as algorithms such as subgroup
11-12
discovery
to reveal patterns in data. The discovery layer will further promote deeper understanding of
the underlying physics. Information from the discovery layer will feed into the decision layer, in which
Bayesian optimization algorithms will decide which data points to sample next and at what level offidelity. Decisions will be translated to queries of the CSD database, a large repository of unlabeled data,and coordinates of structures pulled out of CSD will be fed back into the electronic structure layer toacquire labels. This process will repeat iteratively in order to refine our models. Structures identified aspromising candidates at any point will be further evaluated using high-fidelity GW+BSE calculations toaccurately predict their electronic and optical properties. Due to the high computational cost of MBPT
calculations of large periodic systems with several hundred atoms, we expect most of the computer time
requested to be spent on BerkeleyGW calculations. The computational cost of DFT and ML calculations
is negligible in comparison. Therefore, the proposed development plan is focused primarily on the
BerkeleyGW code. Additional goals are scalability improvements of SISSO and development of Python
workflow management tools to integrate and automate the whole workflow.

§ AppInt funding for Argonne staff for Aurora:– ALCF working with 15 ECP AD so far

Argonne-Intel Center of Excellence – dedicated Intel staff
2e.i. Application Software Requirements

Quantum ESPRESSO:
Language: FORTRAN-90
Libraries: ScaLAPACK, ELPA, LAPACK, BLAS, FFTW
Parallelism: MPI, OpenMP
Past DOE Readiness Programs: NESAP for Cori
BerkeleyGW:
Language: FORTRAN 2008
Libraries: ScaLAPACK, ELPA, LAPACK, BLAS, FFTWIO Libraries: Parallel HDF5
Past DOE Readines Programs: NESAP For CoriSISSO:
Language: FORTRAN
Libraries: BLAS, LAPACK
Parallelism: MPI
Past DOE Readines Programs: optimization on Theta is underway within INCITE project
Multi-fidelity Bayesian Optimization:
Language: Python
Libraries: cuDNN, CUDA, Tensorflow, Pytorch
Parallelism: CUDA on GPUs

http://esp.alcf.anl.gov

5Tracking Aurora Applications Development§ Steps in application preparation– Implementation of science and algorithms– Porting to Aurora programming models– Testing with Aurora SDK on Aurora testbeds– Tuning for performance on Aurora testbeds– Scaling across the Aurora system§ ALCF and Intel working with over 40
projects to ready codes for Aurora§ Effort from over 60 Argonne & Intel
people and numerous outside teams

Application Science Implementation
40
35
30

27

29 28

25

32
25

Port to Aurora Programming Models
34

36

25

27 28

10

2322

2222
181819

20

20
15

30

1515

15
10

8

9

12
8

10

8

10
6

5

5

5

0

4 3 4

0
Partial
20Q4

21Q1

21Q2

Full
21Q3

21Q4

22Q1

22Q2

1 0

22Q4

20Q4

21Q1

11

4 3 3

Early/None
22Q3

26

161515

1312
9

6

24

21Q2

Partial
21Q3

21Q4

Full
22Q1

22Q2

22Q3

22Q4
6

2022 Q4 Aurora Applications Status
Application

Q4

Application

Q4

HACC

GENE

OpenMCBerkelyGW

XGC

Latte

QMCPack

cctbx

AMRWind

MILC

NAMD

NWChemEx

Running

QUDA

MadGraph

Running

LAMMPS

Grid

Running

NekRS

GAMESS

Partially Running

SW4

NYX

FloodFillNetwork

DarkSkyMining

Data Driven CFD

UintahHarveyNalu-Wind

PHASTA

GEM

MFIX-Exa

mb_aligner

FushionDL

RXMD-NN

DCMeshFlow Based Generative Model

E3SM-MMF

FastCaloSim

CANDLE/UNO

spiniFEL

Thornado

Multi-Grid Parameter Opt.

ChromaPorting in Progress

7

Exascale Programming§ We’ve been developing for future target exascale architectures for years– GPU acceleration§ Performance portability across Intel, AMD, NVIDIA– Pick a portability layer (Kokkos, SYCL, OpenMP 5, your own library, TensorFlow, PyTorch)– Work with implementers of layer on target systems§ Lingua Franca– Simulation – compiled languages; math libraries; SYCL, CUDA, HIP, (Kokkos, Raja)– Data & Learning – Python frameworks§ Aurora Programming§ Aurora Development Hardware– Simulation– In the beginning, Xeon integrated graphics• oneAPI, oneMKL– Sequence of pre-production Intel discrete• ECP E4S
GPUs
– Data & Learning– Sunspot• Frameworks backed by oneDAL, oneCCL• 128 nodes of Aurora hardware• DAOS
8

XGC Early Science Project (Simulation; PI: CS Chang)
ITER Tokamak§ Hot exhaust such as He particles
hits the divertor§ Knocks out Tungsten atoms(sputtering)§ Some are ionized and re-enter
the fusion plasma§ Interacts, impacts turbulence andother plasma behaviors§ Will it negatively impact
confinement or energy
production?
§ Early Science campaign:
Predict ITER plasma behavior
with Tungsten impurity ions

Divertor
Tungsten
9

XGC Early Science Project§ XGC gyrokinetic PIC (Particle-in-Cell) fusion plasma physics simulation on unstructured grid
– Multi-ion-species§ Aurora implementation: Kokkos library with SYCL execution space– Portable across {Intel, AMD, NVIDIA} GPUs

Single GPU Performance FOM
Field
solve
Chargescatter§ SimpleFOM =1/(loop_time/(#particles × #timesteps))
Electron push(x60)
Ion push

Collisions
SourcesDiagnostics

Transfer particle
data between
compute nodes

Argonne POCs: T. Williams, A. Scheinberg
Intel POC:
R. Bustamante§ 2 MPI ranks§ XGC1 small test case§ 80 million particles
GPU

SimpleFOM

NVIDIA A100 (Polaris)

1.4×106

Intel® Data Center GPU Max
Series (Sunspot)

2.2×10610

Cancer Metastasis Early Science Project (Data: PI: Amanda Randles)§ ESP GOAL: Investigate circulating tumor cells inhuman vasculatory system– 3D Lattice Boltzmann CFD– FEM for cells, deformation– Immersed boundary method to couple§ In situ exascale data visualization & analysis§ Aurora implementation: bake-off between– SYCL (oneAPI DPC++)– Kokkos with SYCL backend§ HARVEY application translated with Syclomatic, runs
on Intel discrete GPUs
§ MiniApp LBM shows Kokkos-SYCL competitive withhandcoded SYCL
Argonne POCs: J. Insley, S. Rizzi, V. Mateevitsi,
G. Liu, S. PatelIntel POC:

V. Madanath(1). Flow inside ahuman aorta.
Ames, Jeff, et al.
"Low-overhead in situ
visualization using
halo replay." 2019
IEEE 9th Symposiumon Large Data
Analysis andVisualization (LDAV).
IEEE, 2019.

11

Data Science and Learning on Aurora
Aurora will provide for a familiar, productive and performant HPC and AI
software stackIntel AI Analytics Toolkit§ Python and Productivity Languages§ Numba, NumPy, etc.
§ JAX and Julia§ Deep Learning Frameworks:§ PyTorch, TensorFlow, Horovod, DDP,
Deepspeed§ Machine Learning§ OneDAL, scikit-learn, XGBoost, etc.

§ Optimized and scalable communication
using OneCCL§ Spark BigData Analytics stack§ Profiling and debugging tools

https://software.intel.com/content/www/us/en/develop/tools/o
neapi/ai-analytics-toolkit.html

Distributed Asynchronous Object Store (DAOS)
q Primary storage system for Aurora
q Offers high performance in bandwidth and IO
operations
q 230 PB capacityq ≥ 25 TB/s

q Provides a flexible storage API that enables new
I/O paradigms
q Provides compatibility with existing I/O models
such as POSIX, MPI-IO and HDF5
q Open-source storage solution

13

Argonne Leadership Computing Facility

AURORA ESP Data and Learning Projects and Methods

Reinforment learning
Clustering
Uncertainty Quantification
Dimensionality Reduction
Reduced / Surrogate Models
Advanced Statistics
Image and Signal Processing
Graph Analytics
Databases
Advanced Workflows
In Situ Viz & Analysis
14

Argonne Leadership Computing Facility

Enabling Connectomics at Exascale toFacilitate Discoveries in Neuroscience
Machine Learning for Lattice Quantum
Chromodynamics
Accelerated Deep Learning Discovery in
Fusion Energy Science

Learning

Regression

Virtual Drug Response Prediction

Many-Body Perturbation Theory Meets
Machine Learning
Exascale Computational Catalysis
Dark Sky Mining
Data Analytics and Machine Learning for
Exascale CFD
In Situ Visualization and Analysis of FluidStructure-Interaction Simulations

Data

Classification

Simulating and Learning in the ATLAS detector
at the Exascale
14

Data from Gregg Wildenberg, Kasthuri Lab, UChicago

CONNECTOMICS DATA-DRIVEN MODELS
Mouse brain: 70M neurons

sectionsectionsectionsamplesectionsectionsection~1cm^3~1mm^325000
40nm sections
1mm x 1mm
(6nm resolution)

Each section
imaged with EM as
N tiles (8 bit)

Sections
stitched
together

80K x 40K pixels

How much image data is 1mm^3 ? 1e15 voxels -> ~1 PB

DATA CHALLENGES IN CONNECTOMICS
Mouse brain: 70M neurons

Human brain: 80B neurons- Segmentation labels
for each voxel (4xvoxel data)- 3D Mesh- Skeleton

tiny brain~1cm^3~1000cm^3

How much image data
is 1cm^3 ? ~1EB

Reconstructed data
will be much larger:

How much image data is
1000cm^3 ? ~1000 EB(6nm x 6nm x 40nm)

The structures are expected to be
used to seed simulations to study
flow in neuro transmitters, in bettermodeling the brain, brain-inspired
computing, among others.

Courtesy: Nicola Ferrier, Tom Uram, Bobby Kasthuri (Argonne & UChicago)

Data from Gregg Wildenberg, Kasthuri Lab, UChicago

CONNECTOMICS PROCESSING

Sections
stitched
togetherAlignsections

Mask out
non-target
objects
17

Segment
targetobjects

RECONSTRUCTING THE BRAIN CONNECTIVITY

Kasthuri et al, Cell 2015

LARGE-SCALE RECONSTRUCTION▪ Inference (and training) has scaled on CPU-based and GPU-based
supercomputers (parallel granularity: overlapping subvolumes)– Achieved million-way concurrency on Theta supercomputer▪ Image stitching and alignment components are being scaled as well toensure a scalable end-to-end pipeline
Exascale Inference Problem:▪ On a single GPU (A100), we achieve ~80 MegaVoxels/hour using 32-bit(There is still room for improvement here)▪ In reduced precision (8-16 bits), we expect ~1 GigaVoxel/hour per GPU▪ 1 PetaVoxel (1mm3) will take ~1M GPU node hours▪ Approximately, 24 hours on a system with 50K GPUs (considering
overlapping subvolumes)▪ For a mouse brain (1cm3), 1 ExaVoxel, we would need ~3 years on an
exascale system
Dong, et al, “Scaling Distributed Training of Flood-Filling Networks on HPC Infrastructure for Brain Mapping”, 2019
IEEE/ACM Third Workshop on Deep Learning on Supercomputers (DLS) at SC19
Vescovi, et al, “Toward an Automated HPC Pipeline for Processing Large Scale Electron Microscopy Data”, 2020
IEEE/ACM 2nd Annual Workshop on Extreme-scale Experiment-in-the-Loop Computing (XLOOP) at SC19• Progress on Sycl-direct nekRS benchmarks, fusing the nested loops
ExaSMR:
NekRS Performance on Intel® Data Center GPU Max Series• the kernel version of best performanceIntel® Data Center GPU Max Series
with Intel oneAPI DPC++ implementation

Full-core configuration on the left rod bundle on

NekRS is an open-source Navier-Stokes solver based on the spectral element method targetingclassical processors and accelerators like GPUs. Developed in 2019, the code uses high-performance
kernels from libParanumal. For API portable programming OCCA is used.

Full-core configuration on the left and a single 17x17 rod bundle on the right.

20

OpenMC performancehttps://docs.openmc.org

Relative OpenMC Depleted Fuel Inactive Batch Performance on HM-Large
Reactor (Higher is better)
2.5§ Monte Carlo particle transport code for exascale computations§ Intel® Data Center GPU Max Series sustains 999k particles/second using
OpenMP Target offload§ >2x performance gain over A100

2.05
2

1.5

1.17

§ Exascale Compute Project Annual Meeting 2022 presentation:§ https://www.alcf.anl.gov/events/2022-ecp-annual-meeting§ International Conference on Physics of Reactors 2022 presentation:§ https://www.ans.org/meetings/physor2022/session/view-976/

1

1

0.5

0
NVIDIA A100
A100
NVIDIA

1 stack
2 stacks
PVC
1 Stack
PVC
2 StackIntel® Data Center GPU Max Series
Near linear scaling from Intel® Data Center GPU
Max Series 1 Stack to 2 Stack

Application Summary: OpenMC is a Monte Carlo particle transport application that has recently been
ported to the OpenMP target offloading programming model for use on GPU-based systems. The Monte
Carlo method employed by OpenMC is considered the "gold standard" for high-fidelity simulation while
also having the advantage of being a general-purpose method able to simulate nearly any geometry or
material without the need for domain-specific assumptions. However, despite the extreme advantages inease of use and accuracy, Monte Carlo methods like those in OpenMC often suffer from a very highcomputational cost. The extreme performance gains OpenMC has achieved on GPUs, as compared to
traditional CPU architectures, is finally bringing within reach a much larger class of problems that
historically were deemed too expensive to simulate using Monte Carlo methods. The leap in
performance that GPUs are now offering carries with it the potential to disrupt a number of engineering
technology stacks that have traditionally been dominated by non-general deterministic methods. For
instance, faster MC applications may greatly expand the design space and simplify the regulation process
for new nuclear reactor designs -- potentially improving the economics of nuclear energy and therefore
helping to solve the world's climate crisis.

21

ExaSky: CRK-HACC Performanceon Intel® Data Center GPU Max Series
CRK-HACC FOM for SYCL on Intel® Data Center Max Series
relative to CUDA on NVIDIA A100
3.00
2.63

•

Original CUDA kernels
translated to SYCL using
SYCLomatic, with the five
most compute-intensive
kernels hand-optimized byIntel performance engineers.

•

Implemented optimizations
included loop restructuring totake advantage of SYCL
subgroup broadcast
performance.

FOM / A100

2.50

•

•

ExaSky project seeks to verify convergence
between grid and particle methods for simulating
gravity and hydrodynamics to resolve cosmologicalstructure formation on exascale systems.
CRK-HACC employs n-body methods for gravity
and a novel formulation of Smoothed Particle
Hydrodynamics.
• SYCL on Intel® Data Center GPU Max Series.
• CUDA on NVIDIA A100 GPUs.

2.00
1.42

1.50
1.00

1.00

0.50
0.00

NVIDIAA100
A100

1 stack
PVC
1 Stack PVC22stacks
StacksIntel® Data Center GPU Max Series

Figure-of-Merit (FOM) measures throughput of force calculations for 33 million
particles on the GPU, including time required for data transfer between hostand device. Observed relative performance between Intel® Data Center GPU
Max Series and NVIDIA A100 is strongly correlated with the expected single
precision floating point throughput for each architecture.

22

QMCPACK: Performance§ QMCPACK, is a high-performance open-source Quantum Monte Carlo(QMC) simulation code. Its main applications are in computing thequantum mechanical properties of materials with benchmark accuracy,
including for energy storage and quantum materials.

QMCPACK Throughput
A100§ QMCPACK uses C++ and OpenMP target offload, plus wrappers around
vendor optimized linear algebra.
§ Benchmark configuration:– Running `dmc-a512-e6144-DU64` problem. This simulates a
supercell of nickel oxide with 6144 electrons and 512 NiO atoms
total.
– PVC: 2 MPI ranks, with one MPI rank, 8 Walkers, 64 GB of HBM
per Tile. Using Intel(R) oneAPI DPC++/C++ Compiler 2022.1.0– A100 (40GB): 1 MPI Rank, 7 Walkers. LLVM15 compiler.
– The Figure Of Merit (FOM) measure is throughput (walker
moves/second). Higher is better.

PVC

0.15

0.1

FOM (walkers/second)
Argonne POCs: A. Benali, Y. Luo, T. Applencourt
Intel POC:
Y. KimWDMApp: XGC Performance
ESP Project PI: CS Chang• Science case: Predict ITER plasma behavior
with Tungsten impurity ions sputtered fromthe divertor• Gyrokinetic particle-in-cell simulation of
tokamak plasma⏤Kokkos/SYCL on Intel GPUs
⏤Kokkos/CUDA on NVIDIA A100 GPUs.

XGC FOM
XGC1 small test case
80M particles, 2 MPI ranks
2500000
2000000
1500000
1000000
5000000Sunspot (Intel® Data Center Max GPU)
Polaris (NVIDIA A100 GPU)
24

THANK YOU

www.anl.gov

25