1DDN Update Oct, 2016
JLUG2016

DataDirect Networks Japan, Inc
Oct, 2016

Nobu Hashizume© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com

2

DDN Japan 20162016年はExaFlops Systemに向けてDDNが提唱するStorage SystemのFirst Versionを日本で提供できた年になりました© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com

3

DDN Japan 2015

ちなみに去年のJLUGではIMEのコンセプトのお話しとSFA14Kのご紹介を話していました© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com

4
DDN END-TO-END DATA LIFECYCLE MANAGEMENTFAST DATA & COMPUTEPERSISTENT DATA & FILE SYSTEM• SPEED UP YOURAPPS 1000 TIMES!

• HYBRID• EDR IB, 100GB
STORAGEETHERNET• FASTEST
OBJ. STORAGE• HIGHESTSECURITY• HYBRID PUBLIC
CLOUD• MIX SSD
AND DISK• LOWESTCOST ARCHIVE• MULTI-SITE• GPFS™ AND
LUSTRE® BRIDGE• ACCELERATE YOUR
LUSTRE & GPFS
• WORLD’S FASTEST
BURST BUFFER• EMBED APPSAND F/S• 60GB/S• PCI-E FABRIC& 6M IOPSSPEEDLIVE ARCHIVE & COLLABORATION• INTELLIGENT• GLOBALTIERINGCOLLABORATION

CLOUD• OPEN SOURCEACCESS© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com

DDN | EXAScaler & Lustre Case Studies
JCAHPC System

5

•
•

•

University of Tokyo &University of Tsukuba
25 PF System with
8208 KNL Nodes
provided by Fujitsu
I/O System by DDN►►►Intel Omnipath
26 PB ExaScaler/Lustre @ 400 GB/sec
1 PB of IME Burst Buffer with NVMe @ 1400
GB/sec© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

DDN.COM

DDN | EXAScaler & Lustre Case Studies
Reedbush Supercomputer System

6••

University of TokyoI/O System by DDN►►

5 PB ExaScaler/Lustre @145.2 GB/sec
200TB of IME Burst Buffer
with NVMe @ 436.2
GB/sec© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

DDN.COM

DDN | EXAScaler & Lustre Case Studies
Kyoto University Supercomputer System

7••

Kyoto University
I/O System by DDN►►

24PB ExaScaler/Lustre @150 GB/sec
230TB of IME Burst Buffer
with NVMe @ 240 GB/sec© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

DDN.COMIMEシステム@Japan

8

システム

IME

ExaScaler(Lustre)

JCAHPC
Oakforest-PACS

物理960TB
25 x IME14K-OPA

物理32PB
10 x SFA14KXE-OPA

東大 Reedbush

物理230TB
6 x IME14K-EDR

物理6.25PB
3 x SFA14KE-EDR

京大 ACCMS2

物理230TB
6 x IME14K-OPA

物理24PB(P1: 16PB, P2: 8PB)
3 x SFA14K-EDR (P1)2 x SFA14KXE-EDR (P2)

合計

物理1.42PB

物理54.25PB (62.25PB)© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com9© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

DDN.COM

Design Criteria
Overcome Intel 20GB/s Bottleneck§§

No Effective Bandwidth Increases since Sandy Bridge
PCIe-3 maximum speed has not increased

Accommodate New Interconnect Technologies§§

Mellanox EDR, Intel Omni-Path, PCIe-3 x16, 100 Gb/s Ehternet, 32 Gb/s FC
Adaptive Motherboard Design to Encompass Multiple Configurations

Integrate Cutting Edge Solid State Technologies§§

12Gb/S SAS Dual Port, 25W PCIe, NVMe Dual Port
Most efficient IOPs, $/capacity, $/performance

Flexible Configurations for Multiple Use Cases

Scale Up

10

DDN | SFA14K

Scale OutDDN NDA CONFIDENTIAL© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

DDN.COMSFA14K
11

Most Versatile Next-Generation Storage Platform

SFA Powered By SSD & Disk
Maximize Bandwidth and IOPS Efficiency• Data analytics acceleration• Large and small file performance• Up to 60GB/s bandwidth• Support for controller clustering

Embedded Solutions
Parallel File System Appliance: [x]Scaler• Run in the controller head
• GPFS or Lustre• Scale IOPS, throughput, capacity• Up to 38GB/s bandwidth via file system§

All in one integrated
design with expansion
capability§

Dual RedundantControllers IntelHaswell / Broadwell
Processor§

72 Drive High Density
2.5” Enclosure§

SSDs become an
Integral part of theplatform§

48 2.5” dual ported
PCIe/SAS drives and 24
2.5” SAS 12 Gb/S dual
ported drives§

Optimized Building
Block for BW or IOPs§

Direct Connection for 10– 8412 12Gb/S 84 drive
Enclosures

IME Application Acceleration
Embedded IME Technology• Lower latency than flash arrays• Application acceleration• IO provisioning• 50-60GB/s of buffer burst speed

DDN NDA CONFIDENTIAL

Front/Side Isometric View

Right Side View

© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

DDN.COM

12

DDN | SFA14K
Next Generation Hybrid Storage Controller

Design Specs
Appliance

4RU Active/Active Dual-Controller- 48x PCIe NVMe/SAS 2.5” SSD Slots- 24x 12Gb/s SAS 2.5” SSD/HDD Slots- Power Fail-Safe

Internal FabricExternal Expansion

Switched 600+ lanes PCIe Gen3
24 Quad 12Gb/s SAS

CPU – Memory
Controller

Intel® Xeon® Processor E5 v3 (Haswell)
Up to 2048GB DDR4 2133 Memory

Platform
Performance

6M IOPS /4U
60 GB/s /4U

Host & Network
Connectivity

32Gb/s FC
EDR IB
100Gig E
Intel OmniPath

Storage Scalability

72 Internal 2.5” & up to 20 direct attached 84
slot 2.5”/3.5” expansion enclosures

External Clustering

Controller Grid Fabric Interconnect ready

F/W and Software
Options

SFAOS w SFX, Grid- & ExaScaler Embedded,
IME, Open Platform

60M IOPS /Rack
600 GB/s /Rack© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

DDN.COM

13© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

DDN.COM

14

What is IME?

IME creates a new applicationaware fast data tier that
resides right between computeand the parallel file system toaccelerate I/O, reduce latency
and provide greater
operational and economic
efficiency© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

DDN.COM

15

Where IME Provides Value

1. MITIGATES POOR PFSPERFORMANCE caused by PFS
locking, small I/O, andmal-aligned, fragmented I/O patterns.
IME “makes bad apps run well” andalso prevents a poor-behaving app
from impacting the entire
supercomputer.

50 GB/s2) PROVIDES HIGHERPERFORMANCE I/O (bandwidthand latency) to the application.
At ISC14, we demonstrated threeorders of magnitude speed-up
due to this high performance tier

25 MB/s
4 GB/sThis is especially valuable to diverse
workload environments and ISV
applications.

3) IME DRIVES SIGNIFICANTLYMORE EFFICIENT I/O TO THE
PFS by re-aligning and coalescing
data within the non-volatilestorage.
At ISC14, we demonstrated twoorders of magnitude speed-up
due to this efficiency

IOR benchmarks indicate a
3x – 20x speedup on I/Os <32KB.

S3D Turbulent Flow Model© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

DDN.COM

16

IME Architecture

POSIX and MPI-IO interfaces
creates erasure coding
manages data to/from app
extreme update rates
decentralised
fault tolerant
scalable

application interfaces

IME Client
Distributed
Hash Table

place data anywhere
dynamic data placement
improve flash lifetime

Log
Structured
Filesystemmanages buffers for IO
replay to PFS
manages fast rebuilds

IME Server
PFS client© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

DDN.COM

17

IME Architecture

POSIX and MPI-IO interfaces
creates erasure coding
manages data to/from app
extreme update rates
decentralised
fault tolerant
scalable

application interfaces

IME Client
Distributed
Hash Table

place data anywhere
dynamic data placement
improve flash lifetime

Log
Structured
Filesystemmanages buffers for IO
replay to PFS
manages fast rebuilds

IME Server

No application changes
required▶ Applications must berecomplied/relinked toutilise IME▶

PFS client© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

DDN.COM

18

IME Architecture

POSIX and MPI-IO interfaces
creates erasure coding
manages data to/from app
extreme update rates
decentralised
fault tolerant
scalable

application interfaces

IME Client
Distributed
Hash Table

place data anywhere
dynamic data placement
improve flash lifetime

Log
Structured
Filesystemmanages buffers for IO
replay to PFS
manages fast rebuilds

IME Server
PFS client

Client buffers IO
fragments and sends to
IME servers▶ also (optionally) creates
parity buffers according
to RAID scheme chosen▶ sends application anddata to IME servers suchthat loss of a server or
SSD does not result indata loss▶ Heuristics in IME clients
can intelligently pre-fetch
IME-resident data toapplications▶

© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

DDN.COM

19

IME Architecture

POSIX and MPI-IO interfaces

application interfaces▶

IME Client▶

Distributed
Hash Table▶

place data anywhere
dynamic data placement
improve flash lifetime

Log
Structured
Filesystem▶

manages buffers for IO
replay to PFS
manages fast rebuilds

IME Server

creates erasure coding
manages data to/from app
extreme update rates
decentralised
fault tolerant
scalable

PFS client▶

DHT is at the core of IME
distributed index
manages locations of files
and objects
Unique routing and data
placement properties
differentiate IME DHT
from other DHTs
Fast O(1) routing
algorithm built on highperformance, noncryptographic hashalgorithms
Load is uniformly
distributed across DHT
nodes© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

DDN.COM

20

IME Architecture

POSIX and MPI-IO interfaces
creates erasure coding
manages data to/from app
extreme update rates
decentralised
fault tolerant
scalableplace data anywhere
dynamic data placement
improve flash lifetime
manages buffers for IO
replay to PFS
manages fast rebuilds

application interfaces

IME Client
Distributed
Hash Table

Data can be placed
anywhere within IME –any server, any SSD▶ IOPs to SSDs areminimised and optimised
for NAND Flash▶

Log
Structured
Filesystem

IME Server
PFS client© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

DDN.COM

21

IME Architecture

POSIX and MPI-IO interfaces
creates erasure coding
manages data to/from appapplication interfaces

IME Client

extreme update rates
decentralised
fault tolerant
scalable

Distributed
Hash Table

place data anywhere
dynamic data placement
improve flash lifetime

Log
Structured
Filesystemmanages buffers for IO replay
to PFS
manages fast rebuilds

IME Server
PFS client

Actively reorganize I/Os tocoalesce small block I/Os
into large chunks andperform full stripealignment before
submission to the backend, Read acceleration isalmost the reverse
process▶ Protect data by
disseminating erasurecoded chunks across the
IME appliance cluster▶ Pre-fetch data into IMEfrom PFS using out-ofband commands and APIs▶

© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

DDN.COM

22

IME Dataflow

1. Application issues
fragmented,
misaligned IO
2. IME clients sendfragments to IMEservers

Application

Application

Application

Application

IME Client

IME Client

IME Client

IME Client

IME
serverIME
serverIME
serverIME
serverIME
server

3. Fragments to IMEservers accessible
via DHT to all clients
4. Fragments to be
flushed from IME areassembled into PFS
stripes
5. PFS receives
complete PFS stripe

Parallel Filesystem© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

DDN.COM

23

IME Erasure Coding▶
COMPUTE NODES

Data protection against IME server or
SSD Failure is optional• (the lost data is "just cache”)

Data buffer
Data buffer▶

Data buffer
Parity buffer

Erasure Coding calculated at the
Client• Great scaling with extremely high clientcount• Servers don't get clogged up

▶

IME

PFS

PFS

PFS

Erasure coding does reduce useable
Client bandwidth and useable IME
capacity:• 3+1: 56Gb à 42Gb
• 5+1: 56Gb à 47Gb
• 7+1: 56Gb à 49Gb
• 8+1: 56Gb à 50Gb

PARALLEL FILE SYSTEM& STORAGE© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

DDN.COM

24•

Data Migration Policy – Egress (to PFS)

Two modes:► Fully dynamic• Data are flushed to PFS as
they arrive in IME• .. Essentially an LRU►

Explicit

Migrationfrom IME to
PFS

Full-stripe write toparallelfilesystem• Files are not migrated to
the PFS by default• Command is required fromuser or scheduler– Shell command interface• These files *may* be
flushed if IME has reachedcapacity
IME Servers

Parallel Filesystem
Storage© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

DDN.COM

25

Data Migration Policy – Ingress (fromPFS)•

Fully parallel► All IME servers participate iningress procedures•

Explicit only in IME 1.0► Shell command interface► Issued by administrator orscheduler•

Reads of non-cached data aretransparently ‘passed
through to PFS► Data is *not* stored on IME
SSD

Migrationfrom PFS to
IMEFull-stripe write toparallelfilesystem

IME Servers

Parallel Filesystem
Storage© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

DDN.COM

26

IME Fault Tolerance•

Across IME servers►*no* RAID within the IME server•

Flexible based on configuration
and workload►
N+M schemes►
Replication

Full-stripe write toparallel filesystem• Small files►

IME Parity
Group

May be disabled• Per-file basis•

Very High Performance►
Parity group may encapsulate
non-contiguous fragments►
fully non-blocking►
No ‘read-modify-writes’ or
network locking required

parityIME Servers© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

DDN.COM

27

27

Benchmark Results© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

DDN.COM

28

ベンチマーク環境
FDR x8x8
FDR x8
SX6036x8
SX6036x8
FDR x8

SX6036x8
FDR x8

SX6036

FDR x 4SB7700EDR x41 x ES14K

FDR x1

SX6036
FDR x8

FC16 x2
4 x IME Mini

ES14K(DDN Lustre Appliance)
4 x vOSS(Virtual OSS)
4 x EDR (1 x EDR per vOSS)
5 x SS8462 Disk Enclosure
400 x HGST 6TB NL-SAS

MDS/MDT
2x E5-2690
128GB Mem
1 x FDR
2 x FC16
20 x 10K SAS
RAID6(8+2) x 2

IME Mini
2 x E5-2650v2
128GB Memory
2 x FDR Infiniband
23 x Samsung SSD 850 PRO(520MB/sec Write, 510MB Read)© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

DDN.COM

29

ハードウェア、ソフトウェア構成▶ Hardware• Lustre OSS/OSTo ES14K (SFA14KE/5 x SS8462, 400 x 6TB NL-SAS)• Mellanox EDR/FDR Switch (2:1 Blocking Configuration)• 1 x Lustre MDS/MDT
o 1 x SuperMicro Server(2 x E5-2690, 128GB Memory, 1 x FDR)o 1 x EF4024(FC16, 10Krpm 600GB SAS x 20, RAID6(8+2) x 2)

• 32 x Lustre Cliento 16 x Dell R630 (2 x E5-2650 v2, 64GB Memory, 1x FDR)o 16 x Dell R620 (2 x E5-2640 v3, 64GB Memory, 1 x FDR)

▶ Software Stack• ES3.0 Beta(Lustre Server)• CentOS6.7 (Client)• Mellanox OFED-3.2-2.0.0• Lustre-2.5.41.ddn11© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

DDN.COM

30

ベンチマーク概要▶ NeRSCの調達ベンチマークルールを使用• http://www.nersc.gov/users/computationalsystems/cori/nersc-8-procurement/trinity-nersc-8-rfp/nersc8-trinity-benchmarks▶ IOR• POSIX I/O file per process(FPP)• POSIX I/O single shared file(SSF)• MPI/IO file per process(FPP)• MPI/IO single shared file(SSF)© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

DDN.COM

IOR (POSIX, Lustre)
32 clients, 512 process, 3.3TB File Size

31

IOR(Lustre, POSIX, SFA14KE, 400 x NL-SAS)
50000
45000
40000

Throughput (MB/sec)

35000
30000
10K

25000

100K
20000

1000K

15000
10000
5000
0
Write

Read

FPP(File Per Process)

Write

Read

SSF(Single Shared File)© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com

IOR (MPIIO, Lustre)
32 clients, 512 process, 3.3TB File Size

32

IOR(Lustre, MPIIO, SFA14KE, 400 x NL-SAS)
50000
45000
40000

Throughput (MB/sec)

35000
30000
10K

25000

100K
20000

1000K

15000
10000
5000
0
Write

Read

FPP(File Per Process)

Write

Read

SSF(Single Shared File)© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com

IOR (POSIX, Lustre and IME)
32 clients, 512 process, 3.3TB File Size

33

IOR(Lustre, POSIX)

IOR(IME, POSIX)

50000
45000

45000

Hardware Limit

35000
30000
10K

25000

100K

20000

1000K

15000

40000
Throughput (MB/sec)

40000
Throughput (MB/sec)

Hardware Limit

50000

35000
30000

5000

5000
Read

Write

Read

FPP(File Per Process) SSF(Single Shared File)FPP I/O Efficiency ~84%(Write) ~90%(Read)
SSP I/O Efficiency ~5%(Write) ~22%(Read)

1000K

15000
10000

Write

100K

20000

10000

0

10K

25000

0
Write

Read

Write

Read

FPP(File Per Process) SSF(Single Shared File)FPP I/O Efficiency ~97%(Write) ~78%(Read)
SSP I/O Efficiency ~85%(Write) ~81%(Read)
Still under optimizations© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com

IOR (MPIIO, Lustre and IME)
32 clients, 512 process, 3.3TB File Size

34

IOR(Lustre, MPIIO)

IOR(IME, MPIIO)

50000
45000

45000

Hardware Limit

35000
30000
10K

25000

100K

20000

1000K

15000

40000
Throughput (MB/sec)

40000
Throughput (MB/sec)

Hardware Limit

50000

35000
30000

5000

5000
Read

Write

Read

FPP(File Per Process) SSF(Single Shared File)FPP I/O Efficiency ~84%(Write) ~90%(Read)
SSP I/O Efficiency ~5%(Write) ~22%(Read)

1000K

15000
10000

Write

100K

20000

10000

0

10K

25000

0
Write

Read

Write

Read

FPP(File Per Process) SSF(Single Shared File)FPP I/O Efficiency ~97%(Write) ~78%(Read)
SSP I/O Efficiency ~97%(Write) ~81%(Read)
Still under optimizations© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.comThank You!
Keep in touch with us

sales@ddn.com

2929 Patrick Henry Drive
Santa Clara, CA 95054@ddn_limitless

1.800.837.2298
1.818.700.4000

company/datadirect-networks© 2014 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com