The Extreme-scale Scientific Software Stack (E4S)
for Collaborative Open Source Software

Michael A. Heroux, Sandia National Laboratories
Director of Software Technology, US Exascale Computing ProjectThe 2nd R-CCS International Symposium
February 18, 2020

ECP Context for E4S

ECP Software Technology (ST) is one of three focus areas
Performant mission and science applications @ scale
Aggressive RD&D
Project

Mission apps &Deployment to DOE
integrated S/W stack
HPC Facilities

Hardware techadvances

Application
Development (AD)SoftwareTechnology (ST)

Hardwareand Integration (HI)Develop and enhance thepredictive capability of
applications critical to the DOE

Deliver expanded and verticallyintegrated software stack toachieve full potential of exascalecomputingIntegrated delivery of ECP
products on targeted systems at
leading DOE HPC facilities

24 applications including
national security, to energy, earthsystems, economic security,materials, and data

70 unique software products
spanning programming models
and run times, math libraries,
data and visualization

6 US HPC vendors focused on
exascale node and systemdesign; application integrationand software deployment tofacilities

3

ECP ST has six technical areas
PAPI SDE Recorder: Residual per Iteration (662-bus: 662 x 662 with 2,474 nonzeros)

4

10

CG
CGS
BICGSTAB

Residual

100

-4

10

10-8

0

Programming
Models & Runtimes•Enhance and get
ready for exascale thewidely used MPI and
OpenMP
programming models(hybrid programming
models, deep
memory copies)•Development of
performance
portability tools (e.g.
Kokkos and Raja)•Support alternate
models for potential
benefits and risk
mitigation: PGAS(UPC++/GASNet),task-based models(Legion, PaRSEC)•Libraries for deep
memory hierarchy
and power
management

100

200

300

400
500
Iteration

600

700

800

Development
Tools• Continued,
multifaceted
capabilities in
portable, opensource LLVM
compilerecosystem tosupport expectedECParchitectures,
including support
for F18• Performance
analysis tools that
accommodate
newarchitectures,programming
models, e.g.,
PAPI, Tau

900

Math Libraries•Linear algebra,
iterative linear
solvers, direct linear
solvers, integratorsand nonlinear
solvers,
optimization, FFTs,
etc•Performance on new
node architectures;
extreme strong
scalability•Advanced
algorithms for multiphysics, multiscale
simulation andouter-loop analysis•Increasing quality,interoperability,
complementarity ofmath libraries

Data and
Visualization• I/O via the HDF5
API
• Insightful,memory-efficientin-situvisualization andanalysis – Data
reduction via
scientific datacompression• Checkpoint
restartSoftware
Ecosystem•Develop features inSpack necessary tosupport all ST
products in E4S, and
the AD projects that
adopt it•Development of
Spack stacks forreproducible turnkey
deployment of large
collections ofsoftware•Optimization andinteroperability ofcontainers on HPC
systems•Regular E4S
releases of the ST
software stack and
SDKs with regular
integration of new
ST products

NNSA ST• Open source
NNSA Software
projects• Projects that haveboth mission roleand open sciencerole• Major technicalareas: Newprogramming
abstractions,math libraries,
data and viz
libraries• Cover most ST
technology areas• Subject to thesame planning,
reporting andreview processes
4ECP Software Technology Leadership Team
Mike Heroux, Software Technology Director
Mike has been involved in scientific software R&D for 30 years. His first 10 were at Cray in the LIBSCI and scalable apps groups. At Sandia hestarted the Trilinos and Mantevo projects, is author of the HPCG benchmark for TOP500, and leads productivity and sustainability efforts for DOE.
Lois Curfman McInnes, Software Technology Deputy Director
Lois is a senior computational scientist in the Mathematics and Computer Science Division of ANL. She has over 20 years of experience in highperformance numerical software, including development of PETSc and leadership of multi-institutional work toward sustainable scientific software
ecosystems.
Rajeev Thakur, Programming Models and Runtimes (2.3.1)
Rajeev is a senior computer scientist at ANL and most recently led the ECP Software Technology focus area. His research interests are in parallel
programming models, runtime systems, communication libraries, and scalable parallel I/O. He has been involved in the development of open source
software for large-scale HPC systems for over 20 years.
Jeff Vetter, Development Tools (2.3.2)
Jeff is a computer scientist at ORNL, where he leads the Future Technologies Group. He has been involved in research and development ofarchitectures and software for emerging technologies, such as heterogeneous computing and nonvolatile memory, for HPC for over 15 years.
Xaioye (Sherry) Li, Math Libraries (2.3.3)
Sherry is a senior scientist at Berkeley Lab. She has over 20 years of experience in high-performance numerical software, including development ofSuperLU and related linear algebra algorithms and software.
Jim Ahrens, Data and Visualization (2.3.4)
Jim is a senior research scientist at the Los Alamos National Laboratory (LANL) and an expert in data science at scale. He started and actively
contributes to many open-source data science packages including ParaView and Cinema.
Todd Munson, Software Ecosystem and Delivery (2.3.5)
Todd is a computational scientist in the Math and Computer Science Division of ANL. He has nearly 20 years of experience in high-performance
numerical software, including development of PETSc/TAO and project management leadership in the ECP CODAR project.
Rob Neely, NNSA ST (2.3.6)
Rob is an Associate Division Leader in the Center for Applied Scientific Computing (CASC) at LLNL, chair of the Weapons Simulation & Computing
Research Council, and lead for the Sierra Center of Excellence. His efforts span applications, CS research, platforms, and vendor interactions.

We work on products applications need now and into the future
Key themes:• Exploration/development of new algorithms/software for emerging HPC capabilities:• High-concurrency node architectures and advanced memory & storage technologies.
• Enabling access and use via standard APIs.
Software categories:• The next generation of well-known and widely used HPC products (e.g., MPICH, OpenMPI, PETSc)• Some lesser used but known products that address key new requirements (e.g., Kokkos, RAJA, Spack)• New products that enable exploration of emerging HPC requirements (e.g., SICM, zfp, UnifyCR)
Example Products

Engagement

MPI – Backbone of HPC apps

Explore/develop MPICH and OpenMPI new features & standards.

OpenMP/OpenACC –On-node parallelism

Explore/develop new features and standards.

Performance Portability Libraries

Lightweight APIs for compile-time polymorphisms.

LLVM/Vendor compilers

Injecting HPC features, testing/feedback to vendors.

Perf Tools - PAPI, TAU, HPCToolkit

Explore/develop new features.

Math Libraries: BLAS, sparse solvers, etc.

Scalable algorithms and software, critical enabling technologies.

IO: HDF5, MPI-IO, ADIOSStandard and next-gen IO, leveraging non-volatile storage.

Viz/Data Analysis

ParaView-related product development, node concurrency.

ECP ST
Subprojects- WBS- Name- PIs- Project
Managers(PMs)

ECP ST
Stats- 33 L4
subprojects- 10 PI/PC same- 23 PI/PC
different

WBS
2.3
2.3.1
2.3.1.01
2.3.1.07
2.3.1.08
2.3.1.09
2.3.1.14
2.3.1.16
2.3.1.17
2.3.1.18
2.3.1.19
2.3.2
2.3.2.01
2.3.2.06
2.3.2.08
2.3.2.10
2.3.2.11
2.3.2.12
2.3.3
2.3.3.01
2.3.3.06
2.3.3.07
2.3.3.12
2.3.3.13
2.3.3.14
2.3.4
2.3.4.01
2.3.4.09
2.3.4.10
2.3.4.13
2.3.4.14
2.3.4.15
2.3.4.16
2.3.5
2.3.5.01
2.3.5.09
2.3.6
2.3.6.01
2.3.6.02
2.3.6.03

WBS Name
CAM/PI
Software Technology
Heroux, Mike, Carter, J.
Programming Models & Runtimes
Thakur, Rajeev
PMR SDK
Shende, Sameer
Exascale MPI (MPICH)
Balaji, Pavan
LegionMcCormick, Pat
PaRSEC
Bosilica, George
Pagoda: UPC++/GASNet for Lightweight Communication and Global Address Space Support Baden, Scott
SICM
Lang, Michael
OMPI-X
Bernholdt, DavidRAJA/Kokkos
Trott, Christian Robert
Argo: Low-level resource management for the OS and runtime
Beckman, Pete
Development Tools
Vetter, Jeff
Development Tools Software Development Kit
Miller, Barton
Exa-PAPI++: The Exascale Performance Application Programming Interface with Modern C++Dongarra, Jack
Extending HPCToolkit to Measure and Analyze Code Performance on Exascale Platforms
Mellor-Crummey, John
PROTEAS-TUNE
Vetter, Jeff
SOLLVE: Scaling OpenMP with LLVm for Exascale
Chapman, Barbara
FLANG
McCormick, Pat
Mathematical Libraries
McInnes, Lois
Extreme-scale Scientific xSDK for ECP
Yang, Ulrike
Preparing PETSc/TAO for Exascale
Smith, Barry
STRUMPACK/SuperLU/FFTX: sparse direct solvers, preconditioners, and FFT libraries
Li, Xiaoye
Enabling Time Integrators for Exascale Through SUNDIALS/ Hypre
Woodward, Carol
CLOVER: Computational Libraries Optimized Via Exascale Research
Dongarra, Jack
ALExa: Accelerated Libraries for Exascale/ForTrilinos
Turner, John
Data and Visualization
Ahrens, James
Data and Visualization Software Development Kit
Atkins, ChuckADIOS Framework for Scientific Data on Exascale Systems
Klasky, Scott
DataLib: Data Libraries and Services Enabling Exascale ScienceRoss, Rob
ECP/VTK-m
Moreland, Kenneth
VeloC: Very Low Overhead Transparent Multilevel Checkpoint/Restart/Sz
Cappello, Franck
ExaIO - Delivering Efficient Parallel I/O on Exascale Computing Systems with HDF5 and Unify Byna, Suren
ALPINE: Algorithms and Infrastructure for In Situ Visualization and Analysis/ZFP
Ahrens, James
Software Ecosystem and Delivery
Munson, Todd
Software Ecosystem and Delivery Software Development Kit
Willenbring, James M
SW Packaging Technologies
Gamblin, Todd
NNSA ST
Neely, Rob
LANL ATDM
Mike LangLLNL ATDM
Becky Springmeyer
SNL ATDM
Jim Stewart

PC

Shende, Sameer
Guo, Yanfei
McCormick, Pat
Carr, Earl
Hargrove, Paul (and PI)
Vigil, Brittney
Grundhoffer, Alicia
Trujillo, Gabrielle
Gupta, Rinku
Tim Haines
Jagode, Heike
Mellor-Crummey, John
Glassbrook, Dick
Kale, Vivek
Perry-Holby, Alexis
Yang, Ulrike
Munson, Todd
Li, Xiaoye
Woodward, Carol
Carr, Earl
Grundhoffer, Alicia
Bagha, Neelam
Grundhoffer, Alicia
Ross, Rob
Moreland, Kenneth
Ehling, Scott
Bagha, Neelam
Turton, Terry
Willenbring, James M
Gamblin, Todd
Vandenbusch, Tanya Marie
Gamblin, Todd
Trujillo, Gabrielle

7

E4S Overview

Delivering an open, hierarchical software ecosystem
Levels of Integration

Product

Source and Delivery

ECP ST Individual Products• Standard workflow• Existed before ECP

•
•
•
•

Group similar products
Make interoperable
Assure policy compliant
Include external products• Build all SDKs
• Build complete stack• Containerize binaries

ST
Products

SDKs

E4S

Source: ECP L4 teams; Non-ECP Developers; Standards Groups
Delivery: Apps directly; spack; vendor stack; facility stack

Source: ECP SDK teams; Non-ECP Products (policy compliant,
spackified)
Delivery: Apps directly; spack install sdk; future: vendor/facility

Source: ECP E4S team; Non-ECP Products (all dependencies)
Delivery: spack install e4s; containers; CI Testing

ECP ST Open Product Integration Architecture
9Delivering an open, hierarchical software ecosystem
Levels of Integration

Product

Source and Delivery

ECP ST Individual Products• Standard workflow• Existed before ECP

•
•
•
•

Group similar products
Make interoperable
Assure policy compliant
Include external products• Build all SDKs
• Build complete stack• Containerize binaries

ST
Products

SDKs

E4S

Source: ECP L4 teams; Non-ECP Developers; Standards Groups
Delivery: Apps directly; spack; vendor stack; facility stack

Source: ECP SDK teams; Non-ECP Products (policy compliant,
spackified)
Delivery: Apps directly; spack install sdk; future: vendor/facility

Source: ECP E4S team; Non-ECP Products (all dependencies)
Delivery: spack install e4s; containers; CI Testing

ECP ST Open Product Integration Architecture10

E4S: Extreme-scale Scientific Software Stack• Curated release of ECP ST products based on Spack [http://spack.io] package manager• Spack binary build caches for bare-metal installs– x86_64, ppc64le (IBM Power 9), and aarch64 (ARM64)• Container images on DockerHub and E4S website of pre-built binaries of ECP ST products• Base images and full featured containers (GPU support)• GitHub recipes for creating custom images from base images• GitLab integration for building E4S images• E4S validation test suite on GitHub• E4S VirtualBox image with support for container runtimes• Docker• Singularity• Shifter• Charliecloud• AWS image to deploy E4S on EC2

https://e4s.io
11Extreme-scale Scientific Software Stack (E4S)• E4S: A Spack-based distribution of ECP ST and related

and dependent software tested for interoperability andportability to multiple architectures• Provides distinction between SDK usability / general

quality / community and deployment / testing goals• Will leverage and enhance SDK interoperability thrust• Oct 2018: E4S 0.1 - 24 full, 24 partial release products• Jan 2019: E4S 0.2 - 37 full, 10 partial release products• Nov 2019: E4S 1.0 - 50 full, 5 partial release products• Feb 2020: E4S 1.1 - 50 full,10 partial release products

e4s.ioLead: Sameer Shende(U Oregon)

12E4S 1.1 Full Release: 50 ECP Packages and all dependencies• Adios• AML• Argobots• Bolt• Caliper• Darshan• Dyninst• Faodel• Flecsi• Gasnet• GEOPM• Gotcha• HDF5• HPCToolkit• Hypre• Kokkos• Legion• Libnrm• Libquo• Magma• Mercury• MFEM• MPICH• MPIFileUtils• Ninja• OpenMPI• PAPI• Papyrus• Parallel
netCDF• PDT• PETSc• Qthreads• Raja• Rempi• SCR• Spack• Strumpack• Sundials• SuperLU• SZ• Tasmanian• TAU• Trilinos• Turbine

Packages installed using Spack• Umpire• UnifyFS• UPC++ Veloc• Zfp

All ST products
will be released
through E4S

Partial (in-progress) Release: ASCENT, Catalyst, Flang, libEnsemble, LLVM, Visit, others

13

ECP SW Technology
Software Architecture –Spack Software
Distribution System

Spack enables Software distribution for HPC• Spack automates the build and installation of scientific software• Packages are templated, so that users can easily tune for the host environment
No installation required: clone and go$ git clone https://github.com/spack/spack$ spack install hdf5

Simple syntax enables complex installs$ spack install hdf5@1.10.5$ spack install hdf5@1.10.5 %clang@6.0$ spack install hdf5@1.10.5 cppflags="-O3 –g3"
$ spack install hdf5@1.10.5 target=haswell$ spack install hdf5@1.10.5 +threadssafe$ spack install hdf5@1.10.5 +mpi ^mpich@3.2• Ease of use of mainstream tools, with flexibility needed for HPC tuning

github.com/spack/spack• Major victories:• ARES porting time on a new platform was reduced from 2 weeks to 3 hours• Deployment time for 1,300-package stack on Summit supercomputer

reduced from 2 weeks to a 12-hour overnight build• Used by teams across ECP to accelerate development

15

Spack is being used on many of the top HPC systems• Official deployment tool for the

U.S. Exascale Computing Project• 7 of the top 10 supercomputers• High Energy Physics community– Fermilab, CERN, collaborators

Fugaku at RIKEN
DOE/MEXT collaboration• Astra (Sandia)• Fugaku

Summit (ORNL), Sierra (LLNL)

SuperMUC-NG (LRZ,
Germany)

Edison, Cori, Perlmutter (NERSC)
16

Spack is used worldwide!

Over 3,400 software packages
Over 2,000 monthly active users
Over 400 contributors (and growing)

Active users of Spack documentation site for one monthhttps://spack.readthedocs.io
17Spack strategy is to enable exascale software distribution
on both bare metal and containers
1. New capabilities to make HPC packaging easy and automated• Optimized builds and package binaries that exploit the hardware• Workflow automation for facilities, developers, and users• Strong integration with containers as well as facility deployments

Spack Packaging

2. Develop exascale enhancements to container runtimes•

Understand and automate performance/portability tradeoffs•

Optimized containers & build automation for exascale HPC centers•

Enable portability from laptops to exascale

Build / Deployment
Automation

3. Outreach to users• Ongoing working groups, Best practice guides• Tutorials, workshops, BOFs for Spack and Containers

4. Collaboration across ECP• Work with HI and other areas to build service infrastructure• Facilitate curation of packages through E4S and facilities• Ongoing ECP user support

Exascale Container
Runtimes

Bare Metal
Exascale Machines
18Spack heavily involved in the
ECP CI project.
• We have added security features to theopen source GitLab product.

– Integration with center identity management– Integration with schedulers like SLURM, LSF...

• We are democratizing testing at Livermore Computing– Users can run tests across 30+ machines by editing a file
– Previously, each team had to administer own servers• ECP sites are deploying GitLab CI for users– All HPC centers can leverage these improvements

GitLab test runners are now
integrated with HPC machines– NNSA labs plan to deploy common high-side CI infrastructure– We are developing new security policies to allow external open

source code to be tested safely on key machines

User commits
to GitLab
19

ECP SW Technology
Software Architecture –SDKs

Software Development Kits (SDKs): Key delivery vehicle for ECP

A collection of related software products (packages) where coordination across package teams improves usability
and practices, and foster community growth among teams that develop similar and complementary capabilities• Domain scopeCollection makes functional sense• Interaction model
How packages interact; compatible, complementary, interoperable• Community policies
Value statements; serve as criteria for membership• Meta-infrastructure
Invokes build of all packages (Spack), shared test suites• Coordinated plans
Inter-package planning. Augments autonomous package planning

ECP ST SDKs: Grouping similar products
for collaboration & usability
Programming Models &Runtimes Core
Tools & Technologies
Compilers & Support
Math Libraries (xSDK)Viz Analysis and Reduction
Data mgmt., I/O Services & Checkpoint/
Restart“Unity in essentials, otherwise diversity”• Community outreach
Coordinated, combined tutorials, documentation, best practices

21

xSDK community policies
xSDK compatible package: Must satisfy
mandatory xSDK policies:
M1. Support xSDK community GNU Autoconf or CMake options.
M2. Provide a comprehensive test suite.
M3. Employ user-provided MPI communicator.
M4. Give best effort at portability to key architectures.
M5. Provide a documented, reliable way to contact the development team.
M6. Respect system resources and settings made by other previously called
packages.
M7. Come with an open source license.
M8. Provide a runtime API to return the current version number of thesoftware.
M9. Use a limited and well-defined symbol, macro, library, and include filename space.
M10. Provide an accessible repository (not necessarily publicly available).
M11. Have no hardwired print or IO statements.
M12. Allow installing, building, and linking against an outside copy of external
software.
M13. Install headers and libraries under <prefix>/include/ and <prefix>/lib/.
M14. Be buildable using 64 bit pointers. 32 bit is optional.
M15. All xSDK compatibility changes should be sustainable.
M16. The package must support production-quality installation compatible
with the xSDK install tool and xSDK metapackage.

https://xsdk.info/policies
Also recommended policies, whichcurrently are encouraged but not required:
R1. Have a public repository.
R2. Possible to run test suite under valgrind in orderto test for memory corruption issues.
R3. Adopt and document consistent system for error
conditions/exceptions.
R4. Free all system resources it has acquired assoon as they are no longer needed.
R5. Provide a mechanism to export ordered list of
library dependencies.
R6. Provide versions of dependencies.

xSDK member package: Must be an xSDKcompatible package, and it uses or can beused by another package in the xSDK, andthe connecting interface is regularly tested
for regressions.
We welcome feedback. What policies
make sense for your software?
22

xSDK version 0.4.0: December 2018https://xsdk.info

Multiphysics Application C
Each xSDK member package uses orcan be used with one or more xSDK
packages, and the connecting interface
is regularly tested for regressions.

Alquimia

Application B

AMReX

hypre

SLEPc

SUNDIALS

PUMI

MFEM

STRUMPACK

Morelibraries

MAGMA
Tasmanian

DTK

PLASMA

December 2018• 17 math libraries• 2 domaincomponents• 16 mandatoryxSDK community
policies• Spack xSDK
installer

Domain components

Libraries

Frameworks & tools

SW engineering

•
•

•
•

•
•

•
•

Reacting flow, etc.
Reusable.

Solvers, etc.
Interoperable.

Doc generators.
Test, build framework.

HDF5
BLAS

PHIST

Trilinos

SuperLU

Omega_hdeal.IIPETSc

PFLOTRAN

More domaincomponents

Application A

Tested on key machines at ALCF,NERSC, OLCF, also Linux, Mac OS X

Productivity tools.
Models, processes.

Moreexternal
softwareImpact: Improved code quality,
usability, access, sustainability
Foundation for work on
performance portability, deeper
levels of package interoperability

Extreme-Scale Scientific Software Development Kit (xSDK)

23

xsdk

py-libensemble

mfem

py-petsc4py

precicedealiisundialsoce

gsl

intel-tbb

eigen

xSDK Member

py-numpy

Dependency

amrex

ginkgosuperlu-distpumigettext

bzip2

expat

sqlitediﬀutils

tarlibbsdlibxml2

xz

phist

matio

suite-sparsestrumpacknetcdf-cbutterﬂypack

hdf5

glm

pﬂotranomega-harpack-ng

tasmanian

netlib-scalapack

magma

openblas

metis

libﬃ

p4est

petsc

parmetispythonslepc

trilinoshypreplasmapy-setuptools

muparsernanoﬂannboost

py-mpi4py

alquimia

cuda

openmpi

cmake

hwloc

openssl

libpciaccesszlib

numactlutil-macros

adol-clibtool

libiconv

automakeautoconf

perl

gdbm

m4

libsigsegv

readlinencurses

pkgconf24

xSDK version 0.5: November 2019
(21 math libs, 2 domain-specific packages)• AMReX• PETSc/TAO• Trilinos• ButterflyPACK• PHIST• Pflotran• DTK• PLASMA• Alquimia• deal.ii• preCICE• Ginkgo• PUMI• hypre• SLEPc• libEnsemble• STRUMPACK• MAGMA• SUNDIALS• MFEM• SuperLU• Omega_h• TasmanianNotes:• Growth:§ 5 in release 0.1.
§ 7 in 0.2§ 9 in 0.3§ 19 in 0.4§ 23 in 0.5• You do not need to build all packages.
• We build and test all packages.
• Any subset is guaranteed to build if using thesame build parameters, platforms.
• Similar builds should work or require less effort
for success.
25

SDK “Horizontal” Grouping:
Key Quality Improvement
DriverPETSc

Trilinos

SuperLU Version X

SuperLU Version Y

Horizonal (vs Vertical) Coupling

Horizontal grouping:• Assures X=Y.
• Protects against regressions.
• Transforms code coupling fromheroic effort to turnkey.

– Common substrate– Similar function and purpose•e.g., compiler frameworks, math libraries– Potential benefit from common Community Policies•Best practices in software design and development and customer support– Used together, but not in the long vertical dependency chain sense– Support for (and design of) common interfaces•Commonly an aspiration, not yet reality

26

Development of xSDK Community Policies• Community policies now available on Github for revision control and to preserve history:
https://github.com/xsdk-project/xsdk-community-policies• Established process on how to change or add policies• Newest release: 0.5.0
Also recommended policies, which currently are encouraged but
not required:
R1. Have a public repository.
R2. Possible to run test suite under valgrind in order to test for memory
corruption issues.
R3. Adopt and document consistent system for error conditions/exceptions.
R4. Free all system resources it has acquired as soon as they are no longer
needed.
R5. Provide a mechanism to export ordered list of library dependencies.
R6. Provide versions of dependencies.
R7. Have README, SUPPORT, LICENSE, and CHANGELOG file in top
directory.
Changes in 0.5.0:
q
New recommended policy R7q
Dropped the requirement to detect MPI 2 features in M3
q
Made various editorial changes in M5, M13, M15, and R2 forclarification or to fix typos.
27 Exascale Computing ProjectIntegration of Community Policies• Potential new xSDK members will fill out
compatibility form:

https://github.com/xsdk-project/xsdk-policy-compatibility• xSDK team will check for compatibility
and approve if compatible• Designing and implementing xSDK policy
checker to automate the policy compliancechecking.
– Implemented checkers for a few policy rules.
Examples:• M3: Employ user-provided MPI Communicator: Policy checker lists the file names that
contain MPI_COMM_WORLD.
• M7: Come with an open source license. Policy checker search license files at the top
directly of the source and scan the headers of source code files.
28 Exascale Computing ProjectProcesses for xSDK release and delivery• 2-level release process– xSDK• Ensure and test compatibility of mostly independent package releases

xSDK delivery process•

Regular releases of softwareand documentation, primarily
through member package
release processes•

Anytime open access to
production software from
GitHub, BitBucket and related
community platforms– xSDK member packages• Achieve compatibility with xSDK community policies prior to release– https://github.com/xsdk-project/xsdk-policy-compatibility• Have a Spack package• Port to target platforms• Provide user support• Obtaining the latest release: https://xsdk.info/releases• Draft xSDK package release process checklist:– https://docs.google.com/document/d/16y2bL1RZg8wke0vY8c97ssvhRYNez34Q4QGg4LolEUk/edit?usp=sharing
29 Exascale Computing Project

WBS Areas

There are 5 L4 projects to define and/or enhance SDKs
• Each L3 area has an L4 project devoted to

SDK definition and coordination across theportfolio• Software ecosystem L4 project focuses on

packaging– Spack– Containers• Strong coordination with HI Software

Deployment projects• Drives milestone-based planning

30

ECP ST SDKs will span all technology areas
Motivation: Properly chosen cross-team interactions will build relationships that support interoperability, usability,
sustainability, quality, and productivity within ECP ST.
Action Plan: Identify product groupings where coordination across development teams will improve usability andpractices, and foster community growth among teams that develop similar and complementary capabilities.
PMR Core (17)Compilersand Support (7)Tools andTechnology (11)xSDK (16)Visualization Analysis
and Reduction (9)Data mgmt, I/O Services,
Checkpoint restart (12)Ecosystem/E4S
at-large (12)QUO

openarc

TAU

hypre

ParaView

SCR

mpiFileUtils

Papyrus

Kitsune

HPCToolkit

FleSCI

Catalyst

FAODELTriBITS

SICM

LLVM

Dyninst Binary Tools

MFEM

VTK-m

ROMIO

MarFS

LegionCHiLL autotuning comp

GotchaKokkoskernels

SZ

Mercury (Mochi suite)

GUFI

Kokkos (support)

LLVM openMP compCaliper

Trilinos

zfp

HDF5

Intel GEOPM

RAJAOpenMP V & V

PAPI

SUNDIALS

VisIt

Parallel netCDF

BEE

CHAI

Flang/LLVM Fortran comp

Program Database Toolkit

PETSc/TAO

ASCENT

ADIOS

FSEFI

PaRSEC*

Search (random forests)

libEnsemble

CinemaDarshan

Kitten Lightweight Kernel

DARMA

SibokaSTRUMPACK

ROVER

UnifyCR

COOLR

GASNet-EX

C2C

SuperLU

VeloC

NRM

Qthreads

Sonar

ForTrilinos

IOSS

ArgoContainers

BOLT

SLATE

HXHIM

Spack

UPC++

MAGMA

MPICH

DTK

Open MPI

TasmanianMath Libraries

UmpireTuckerMPI

Data and Vis

AML

PMR
Tools

LegendEcosystems and delivery

Data SDK: HDF5 API Initiative•

Adding The HDF Group (THG) to the Data & Vis SDK project to assist in various aspects ofimproving the use of the HDF5 API–

Support and training•–

Compatible APIs with other I/O libraries•••

Direct support and training will be provided to ECP applications and ST projects to assist in theperformant use of HDF5
DataLib (pnetcdf) and ADIOS will develop interfaces compatible with the HDF5 API, possibly through the
HDF5 Virtual Object Layer
THG will provide support to the I/O projects to assist in achieving this compatibility

Both Kitware and THG will identify the I/O usage patterns w.r.t. metadata operations, bulk data
operations, access patterns.
––

This will help identify usage patterns that can be improved on by applications and ST projects
Also will help identify usage patterns that HDF5 can work to optimize for.

32

SDK Summary• SDKs will help reduce complexity of delivery:– Hierarchical build targets.
– Distribution of software integration responsibilities.

• New Effort: Started in April 2018, fully established in August 2018.
• Extending the SDK approach to all ECP ST domains.
– SDKs create a horizontal coupling of software products, teams.
– Create opportunities for better, faster, cheaper – pick all three.

• First concrete effort: Spack target to build all packages in an SDK.
– Decide on good groupings.
– Not necessarily trivial: Version compatibility issues, Coordination of common dependencies.

• Longer term:– Establish community policies, enhance best practices sharing.
– Provide a mechanism for shared infrastructure, testing, training, etc.
– Enable community expansion beyond ECP.
33

Putting it all together: Products + SDKs + E4S• In ECP, teams increasingly need to ensure that their libraries

Integrateand components work together–

E4S

Historically, HPC codes used very few dependencies

ECP-wide
software release
https://e4s.io• Now, groups of teams work together on small releases of“Software Development Kits”• SDKs will be rolled into a larger, periodic release.
DeliverDevelop

DeliverMath
Libraries
Test

Visualization
Package

Build

Develop

Test

Package

Build

Build

Deliver
Deliver

Test

Develop

Programming…
Models
Test
Package
Build

34

E4S Collaborations

NSF Collaborations

https://oaciss.uoregon.edu/NSFDOE19/agenda.html

36Extending Collaborations• E4S/SDK collaborations make sense with:– HPC open source development projects:• deal.II (NSF math library),• PHIST (DLR Germany library).

– Commercial open source packagers/distributors:• OpenHPC.
• HPC systems vendors.
– HPC systems facilities:• SDSC, TACC, others.
– Other organizations in search of collaborative open source software foundation:• NSF science teams.
• NOAA, others.
• International collaborators.
37

E4S: Building on top of previous efforts• E4S did not emerge from nothing.
• Leveraging the work of many others.
• HPC Linux: Work done at U of Oregon, and at ParaTools.
• IDEAS-Classic: xSDK – the original SDK continuing under ECP.
• Spack – Pre-dates E4S.

38

E4S: ARM64 Base Container Images• Dockerhub• Support for

NVDIA GPUs

39

E4S Wrap Up

Delivering an open, hierarchical software ecosystem
Levels of Integration

Product

Source and Delivery

ECP ST Individual Products
Source: ECP L4 teams; Non-ECP Developers; Standards Groups
Delivery: Apps directly; spack; vendor stack; facility stackST
Products• ECP Legacy: An open, hierarchical and extensible HPC software ecosystem• Standard workflow• Existed before ECP

•
•
•
•

• SDK
communities
for enhancedSource:
developer
experienceGroupsimilarproducts
ECPandSDKuser
teams;
Non-ECP Products (policy compliant,
spackified)Makeinteroperable• Easydeployment via source and by container technologies
Delivery: Apps directly; spack install sdk; future: vendor/facility
Assure policy compliant
SDKs• Inexternal
collaboration
with DOE facilities, vendors, other US agencies and strategic collaborators
Includeproducts• A model for post-ECP coordinated software development and delivery.

• Build all SDKs
• Build complete stack• Containerize binaries

E4S

Source: ECP E4S team; Non-ECP Products (all dependencies)
Delivery: spack install e4s; containers; CI Testing

ECP ST Open Product Integration Architecture41

E4S Summary

What E4S is not

What E4S is• Extensible, open architecture software ecosystem• A closed system taking contributions only from DOE

software development teams.

• A monolithic, take-it-or-leave-it software behemoth.

• A commercial product.

accepting contributions from US and international teams.

• Framework for collaborative open-source productintegration.

• A full collection if compatible software capabilities and• A manifest of a la carte selectable software capabilities.

• Vehicle for delivering high-quality reusable software

products in collaboration with others.

• The conduit for future leading edge HPC software• A simple packaging of existing software.

targeting scalable next-generation computing platforms.

• A hierarchical software framework to enhance (via SDKs)

software interoperability and quality expectations.
42Final Remarks• Software Development Kits (SDKs) provide a new software coordination mechanism:– Do not significantly increase the number of software products in the HPC ecosystem.
– Provide intermediate build, install and test targets for E4S (reducing complexity).
– Enable development teams to improve and standardize look-and-feel, best practices, policies.
– Improve interoperability, interchangeability of similar products.
– Fosters communities of developers in a cooperative/competitive environment.
– Provides integration point for SDK-compatible, non-ECP products.

• The Extreme-scale Scientific Software Stack (E4S) provides a complete HPC software stack:– Does not significantly increase the number of software products in the HPC ecosystem.
– Provides a coordinated approach to building, installing and testing HPC software.
– Tests some products with a subset of configurations on a subset of platforms.
– Improves stability of the ST stack so that any subset is more stable.
– Provides a complete software stack, including non-ECP products.

• The SDK/E4S architecture is open:– Enables light-weight coordinated collaboration among open source software teams.
– ECP seeks collaboration: libraries/tools and SDKs, facilities and E4S.
43

ECP Software Technology Capability
Assessment Report (CAR) Version 2.0• Comprehensive document about ECP

ST structure, progress and planning.

• Version 2.0 – extensive revision:– FY20 – 23 Organization.
– Planning, Execution, Tracking &Assessment processes.
– E4S/SDK details.
– 2-page writeups for each productdevelopment effort.
– Released February 1, 2020.
https://exascaleproject.org/wp-content/uploads/2020/02/ECP-ST-CAR-V20-1.pdf
44