LustreFutures for HPC Storage

Peter Braampeter@braam.io
2017-10

Contents• Lustre 1998 - 2017
• Storage Tier Hardware Technology• IO in big US HPC Systems• Using IO in HPC• New Software Developments• Challenges and Conclusion
Speaker: introduced Lustre and other ideas. Presently independent
researcher. Focus on future I/O, SKA telescope.
1/11/17

2

Lustre 1998 - 2017

1/11/173

A few thoughts & facts
Lustre has delivered:
1. interesting work to many 100’s of people
2. business to dozens of companies
3. a cornerstone to HPC infrastructure
The commitment of the user, business and developer community toLustre has created this value. I hope everyone will experience a senseof purpose they have contributed.
This community has shaped my life – completely unexpectedly, beyond
my wildest dreams. Thank you!
1/11/174

What wasn’t done?

The Lustre object servers could have shaped the cloud
The metadata approach was too traditional
However: this was a mandate to focus
this focus was key to Lustre’s success

1/11/175

Missed opportunities
A complete re-write:
in a modern languagewith user space servers
new abstractions (containers, write back caches)
100x lower maintenanceGood usability:
dozens of nearly useless GUI’s
impossibly difficult configuration and tuning
only highly skilled vendors deliver great Lustre systems

1/11/176

Future of HPC Storage

1/11/17

7

Storage Tiers

1/11/178

Tier Technologies and ParametersHigh
BandwidthMemory

RAM

NVRAM
XPOINT / PCM /STTRAM

FLASH

DISK

TAPE

BW Cost$/ (GB/s)$12$10>$10$200

$2K

$20K

Capacity Cost
$/GB

$9.60

$8

<$8$0.3

$0.02

$0.01

Node BW(GB/sec)

1 TB/s100 GB/s<100GB/sec

20 GB/s

5 GB/sCluster BW(TB/sec)

10-100 PB/s

100 TB/s<100TB/sec

5 TB/s100 GB/s

10’s GB/s

Software

HDF5DAOS

DDN IME
Cray Data Warp
LustreLustre / GPFS
Campaign Storage

Archive &Campaign
Storage

1/11/17

9Economy

Modeling:

birth offlash
cache

cost of TB/s
bandwidthcluster wait time for IO = O (1 / bandwidth)storage bandwidth cost = O (1/capacity cost)
cost variations: 10^2-10^3 x
perf variations: 10^2-10^3 x
many new models possible: tape - nvram

diskbirth ofsolidstatecapacity
tierflash2013

cost ofcapacity

diskflash2022 (?)

1/11/1710

Large US Deployments

1/11/1711

Large US deployments 2000 - 2023
Server Centric Storage

Client Centric Storage

2000 – 20142020 (Capability System at LANL)

2014-20182023 (US Exascale)

Racks with compute nodes
Disk enclosures
up to 1TB/sec, disk, 100x size of RAM
Lustre / GPFS

Racks with bladesFlash caches: 10x RAM, 5 TB/sec
Lustre / GPFS / DDN IME / Cray
Secondary disk FS: 100x RAM, 1TB/sec
Clients for many core Intel chips– notyet for GPUs

1/11/17

Every compute node: Lustre ZFS flash server
100 TB/sec (10 GB/sec /node)
Secondary disk storage (100PB) – Campaign
Storage

1TB HBM / node Object store: 10,000 NVRAM server nodes
1PB / sec (100GB/sec / node)secondary flash(?) storage(1EB)

12

Using IO in HPC

1/11/17

13

Cluster File System Performance Trouble
Massive exchange of small data
Not un-common

Root causes:
1. Concurrent resources required
2. Data layout must be carefully chosen• Ideally 1 process uses only 1 server• Reasonable stripe sizes

3. Complicated metadata data interactions• 2007: ADIOS library addresses these issues
1/11/1714

What does ADIOS really do?

1/11/17

What needs to be written

How will it be written?

• New API – not POSIX, verysimple• Form group of processes• Declare what items and how
many need to be read / written• Do IO asynchronously• External specification of file
• Use software plugin to drive theright storage infrastructure• Describe the desired layout ofdata

15

Support for storing structured data
HDF5
• HPC standard for arrays, KV store and more
• Surprisingly small overlap with similar custom data layout for cloud• Other formats (e.g. NetCDF) starting to leverage HDF5
• HDF5 beginning to use sophisticated lower layers (e.g. ADIOS)
Desired for Lustre: Very best HDF5 integration.
1/11/17

16

New IO Software

1/11/17

17

DAOS – distributed async object store
A USA DOE – Intel – HDF5 group collaboration2012 – 2015: initial prototype based on Lustre / ZFS
2015 - : 2nd pre-production NVM implementation
Open Source
Key capabilities:• Low level NVM transactional, versioned storage• Distributed groups of processes collaborate on IO• Scales to 100K’s servers, 1B client processes• Redundancy
Applications• Underpinning for HDF5 and legacy file system

HDF5 and File System Drivers

Transactional distributed
object interface withredundancy

Distributed NVM
object store
containercontainer
targets(devices)

targets(devices)

targets(devices)

NVM object store

Probably not so easy to use directly

1/11/1718

Role of containers
Fundamental issue: fast sidevs slow side in hierarchy
Hence:- Create fine grained data
- Move coarse grained data
Container implementations- Can be based on ZFS- Respect slower and fasterinterfaces

Fast Tier:applicationinterface
Container layer

Layer 1

Base layer

Other approaches:- DDN IME
- Cray Data Warp

1/11/17

Slower tier serialize

Implementation

ZFS clone
ZFS filesystem

ZFS clone

Serialized
differential

Analytics
differential

ZFS
snapshot

Serialized
differential

Container
analytics

ZFS
snapshot

ZFS Pool

19Challenges &Conclusions

1/11/1720

Challenges

1/11/17

API introduction

Deployment contrast• Cluster File Systems leveraged
well established API: POSIX• New systems must create andestablish API. “All” applications
must come along.

• HPC must become more cloudcompatible• Cloud Data Storage presently has
fundamentally different qualities

21

Conclusions
Beauty and Simplicity• Simple, convincing systems are emerging:• DAOS, ADIOS, Containers, HDF5, Campaign Storage• Exciting challenges exist• Hardware developments have been fantastic

1/11/1722

Thank you

1/11/1723