(SOME) EUROPEAN EXASCALE PROJECTS
Adrian Jackson@adrianjhpc
a.jackson@epcc.ed.ac.uk

Exascale I/O with Storage
Class Memory

Warning!
• Terminology will be annoying:• NVDIMM• NVRAM• SCM• ……

• My fault, but people argue which is the mostappropriate• So using them all to annoy as many people aspossible NEXTGenIO summary

Project• Research & Innovation
Action• 36 month duration• €8.1 million• Approx. 50% committed
to hardware developmentPartners• EPCC• INTEL• FUJITSU
• BSC
• TUD
• ALLINEA• ECMWF• ARCTUR

New Memory Hierarchies• High bandwidth, on processor memory• Large, high bandwidth cache• Latency cost for individual access may be anissueCacheMemoryStorage• Main memory• DRAM• Costly in terms of energy, potential for lower
latencies than high bandwidth memory• Storage class memory• High capacity, ultra fast storage• Low energy (when at rest) but still slower
than DRAM• Available through same memory controller
as main memory, programs have access tomemory address spaceCache
NVRAM
HBW MemoryMemory
NVRAM
NVRAMSlowStorage
Fast Storage
Slow Storage

Non-volatile memory• Non-volatile RAM• 3D XPoint technology• STT-RAM• Much larger capacity than DRAM
• Hosted in the DRAM slots, controlled by a standard
memory controller• Slower than DRAM by a small factor, but
significantly faster than SSDs
• STT-RAM• Read fast and low energy• Write slow and high energy• Trade off between durability and performance• Can sacrifice data persistence for faster writes

SRAM vs NVRAM• SRAM used for cache• High performance but costly• Die area
• Energy leakage• DRAM lower cost but lower performance• Higher power/refresh requirement• NVRAM technologies offer• Much smaller implementation area
• No refresh/ no/low energy leakage• Independent read/write cycles• NVDIMM offers• Persistency• Direct access (DAX)NVDIMMs• Non-volatile memory already exists• NVDIMM-N:• DRAM with NAND Flash on board• External power source (i.e super capacitors)• Data automatically moved to flash on power failure with capacitor support, moved
back when power restored• Persistence functionality with memory performance (and capacity)• NVDIMM-F:• NAND Flash in memory form
• No DRAM• Accessed through block mode (like SSD)

• NVDIMM-P:• Combination of N and F• Direct mapped DRAM and NAND Flash• Both block and direct memory access possible• 3D Xpoint, when it comes••••

NVDIMM-P like (i.e. direct memory access and block)
But no DRAM on board
Likely to be paired with DRAM in the memory channel
Real differentiator (from NVDIMM-N) likely to be capacity and cost

Memory levels• SCM in general is likely to have different
memory modes* (like MCDRAM on KNL):• Two-level memory (2LM)
SCM

Processor

Cache

MemoryRegions

OS Main Memory

DRAMSCM• One-level memory (1LM)

ApplicationDirectRegions*https://www.google.com/patents/US20150178204

DRAM
MemoryOS Main Memory

Processor

9Storage Class Memory• The “memory” usage model allows for theextension of the main memory• The data is volatile like normal DRAM based mainmemory• The “storage” usage model which supports theuse of NVRAM like a classic block device• E.g. like a very fast SSD• The “application direct” (DAX) usage modelmaps persistent storage from the NVRAM
directly into the main memory address space• Direct CPU load/store instructions for persistent
main memory regions

I/O

I/O Performance• https://www.archer.ac.uk/documentation/white-papers/parallelIObenchmarking/ARCHER-Parallel-IO-1.0.pdf

ARCHER workload

Burst Buffer• Non-volatile already becoming part of HPC hardware
stack• SSDs offer high I/O performance but at a cost
• How to utilise in large scale systems?

• Burst-buffer hardware accelerating parallel filesystem• Cray DataWarp• DDN IME (Infinite Memory Engine)Burst buffer
external filesystemburstfilesystem

external filesystemhigh performance networkhigh performance network

compute nodes

compute nodes

Moving beyond burst buffer• Non-volatile is coming to the node rather than thefilesystem• Argonne Theta machine has 128GB SSD in eachcompute node
• And lustre
external filesystemhigh performance network

compute nodes

I/O application patterns

Individual I/O
Operation

I/O Runtime
Contribution

Enabling new I/O
0.4
I/O time
Fraction of runtime spent on I/O

0.35
0.3
0.25
0.2
0.15
0.1
0.05
0Lustre End Lustre Every Optane End
IterationOptaneEveryIterationSSD End

SSD Every
Iteration

Mem End Mem Every
Iteration

Exploiting distributed storage

Programming SCM• Block memory mode
• Standard filesystem api’s• Will incur block mode overheads (not bytegranularity, kernel interrupts, etc…)• App Direct/DAX mode
• Volatile memory access can use standard
load/store• NVM library• pmem.io• Persistentload/store• memory
mapped file
likefunctionality

Using distributed storage• Without changing applications• Large memory space/in-memory database etc…
• Local filesystem• Users manage data themselves• No global data access/namespace, large number of files• Still require global filesystem for persistenceUsing distributed storage• Without changing applications• Filesystem buffer• Pre-load data into NVRAM from filesystem• Use NVRAM for I/O and write data back to filesystem atthe end• Requires systemware to preload and postmove data
• Uses filesystem as namespace manager

Using distributed storage• Without changing applications• Global filesystem• Requires functionality to create and tear down global
filesystems for individual jobs
• Requires filesystem that works across nodes• Requires functionality to preload and postmove filesystems• Need to be able to support multiple filesystems acrosssystem

Using distributed storage• With changes to applications• Object store• Needs same functionality as global filesystem• Removes need for POSIX, or POSIX-like functionality

Using distributed storage• New usage models• Resident data sets
• Sharing preloaded data across a range of jobs
• Data analytic workflows• How to control access/authorisation/security/etc….?

• Workflows• Producer-consumer model• Remove filesystem from intermediate stages

Using distributed storage• Workflows• How to enable different sized applications?

• How to schedule these jobs fairly?
• How to enable secure access?

The challenge of distributedstorage• Enabling all the use cases in multi-user, multi-job
environment is the real challenge• Heterogeneous scheduling mix• Different requirements on the SCM• Scheduling across these resources• Enabling sharing of nodes• Not impacting on node compute performance• etc….

• Enabling applications to do more I/O• Large numbers of our applications don’t heavily use
I/O at the moment• What can we enable if I/O is significantly cheaperPotential solutions• Large memory space• Burst buffer• Filesystem across NVRAM in nodes• HSM functionality• Object store across nodes• Checkpointing and I/O libraries

NEXTGenIO Systemware

Compute node systemware

User node systemware

Summary• Storage class memory is coming• Price and capacity remains to be seen, but initial
indications are interesting (large, cheaper than DRAM
on a per GB)• In-node persistent storage likely to come to (maybesome) HPC and HPDA systems shortly• Applications can program directly but….
• …potentially systemware can handle functionality for
applications, at least in transition period• Interesting times• Convergence of HPC and HPDA (maybe)• Different data usage/memory access models may
become more interesting• Certainly benefits for single usage machines, i.e.
bioinformatics, weather and climate, etc…

DEEP-EST: Extreme Scale
Projects

DEEP-ESTDEEP-EST• EU Horion2020
funded: 754304• €15M 3 year project• Create a firstversion of theModular
Supercomputer
Architecture (MSA)
defined in DEEPand DEEP-ER• 15 partners

DEEP-ESTSoftware integration• Schedule across heterogeneous hardwaresetup• Monitor power, energy, and performance• Support single filesystem and storage
target from different modules• Develop/extend programming
environments to use new modules– Data analytics modules
DEEP-ESTNetwork AttachedMemory• NAM is a physical circuit board with memory/storage, network
interfaces, and an FPGA
• A NAM is treated as regular node
– Attaches to the same network(as any other node)• Every process in the system can• Write to it• Read from it• Use application specific functions• Access rights and security managed by acentral instance: NAM Manager

DEEP-ESTDEEP-ER NAM• Xilinx Virtex 7 FPGA
• Hybrid Memory Cube (HMC)– DRAM based– 2 GB capacity– 40 GB/s read/write• 2* Network 12x connections– ~20GB/s read/write in DEEP-ER
Checkpoint

DEEP-ESTRestartDEEP-EST NAM
Characteristics• Memory/Storage directly connected
to network through FPGA
• NAM is integrated in the Fabri³• Per NAM:
– 16 GB/s write + 16 GB/s read
– Up to 64 TByte persistent storage(NAND, 3DXpoint)– (up to 32 GB DDR4)• Fabri³– Maximum 8 NAMs per Fabri³– 128 GB/s write + 128 GB/s read
– 512 TByte persistent storage• Lower latency and higher bandwidththan writing to or reading from a
regular remote node memory or thePFSDEEP-ESTPotential DEEP-ESTNAM Use cases• Checkpointing target– Use existing libNAM functions (e.g. with SIONlib)• Parallel File System– Use file operations (fopen, fwrite, …) as if theNAM was the PFS– Use as intermediate file-system as BeeOND
extension• Burst buffer alternative• Fast, global shared memory• MPI one-sided sharedDEEP-EST

Global CollectiveEngine• Accelerate collectivecalls inside thenetwork• Use the same
physical board as
the NAM• Use FPGA toimplement
collectives
DEEP-ESTGCE• Jobs register groups of participants for acollective group– In MPI context this reflects generation of acommunicator• Jobs de-register groups of participants– In MPI context this reflects destroying acommunicator• Registering a collective group will trigger
allocation of resources on the GCE– E.g. State & scratchpad memory allocation
DEEP-ESTGCE• Jobs send request messages describing
the collective and their parameters– Source & destination buffers, size, operandtype, collective operation, compute operationin case of reduce,…

• GCE performs the actual work completely
asychronously to the actual compute job• GCE distributes data back to theparticipating processes and notifies them
DEEP-ESTGCE Benefits• Fully concurrent to actual compute code
• Supports the idea of non-blocking
collectives ideally• Optimize bandwidth requirements in thenetwork• Reduces latency, since no synchronousintermediate full stack send & receive
messages to be processed by computenodes
DEEP-EST