1DDN Update for 2017
PCCC

DataDirect Networks Japan, IncDec, 2016

橋爪信明

© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com

2DDN END-TO-END DATA LIFECYCLE MANAGEMENT

Parallel File System
HDD/SSDLIVE ARCHIVE,
SHARING & CLOUD

FlashScale
FlashScale

BURST &COMPUTE• 1000X APP &FILE SYSTEM
SPEED UP• FASTEST, LOW
LATENCY &BEST COST SSD• MOST VERSATILEEMBEDDED FILE &BLOCK STORAGE• FASTEST,SCALABLE OBJECTSTORAGE• PREDICTIVE
BURST
BUFFER• 600GB/S & 60M
IOPS,
5+PB/RACK• EDR IB, 100 GbE,Omnipath, PCI-E• GLOBAL MULTI-SITE
SECURE TIERING• OPEN SOURCE &HYBRID CLOUD• BRIDGE S3,SWIFT, GPFS™ &LUSTRE®

© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com

3

Why IME?
cater better for real-world IO patterns

IO Benchmarkingfor acceptance© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com

4

Why IME?
cater better for real-world IO patterns

All the time after that
4
© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com

DDN のFlash(IME)に対する取り組み

5

従来のHPC向け並列ファイルシステムでは解決出来ないIOワークロード（Big Data, AIなど）に向けてFlashを活用した新しいIOシステムを提案►2014 : Infinite Memory Engine(IME)発表►2015 : IMEを大規模スパコンに提案►

2016 : IMEを国内3サイトに提供►

2017 : 小さいIMEを展開する予定© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com

6DDN | IMEApplication I/O Workflow
ComputeDiverse, highconcurrency
applications

Lightweight IME clientintercepts application I/O.
Places fragments into buffers+ parity

Fast Data
NVM & SSD

IME client sends
fragments to IMEservers

IME servers write
buffers to NVM andmanage internal
metadata

Persistent
Data (Disk)IME servers writealigned sequential
I/O to SFA backend

Parallel File system
operates atmaximum efficiency© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com

DDN | EXAScaler & Lustre Case Studies
JCAHPC System7•••

University of Tokyo &University of Tsukuba
25 PF System with
8208 KNL Nodes
provided by Fujitsu
I/O System by DDN►►►Intel Omnipath
26 PB ExaScaler/Lustre @ 400 GB/sec
1 PB of IME Burst Buffer with NVMe @ 1400
GB/sec© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

DDN.COM

DDN | EXAScaler & Lustre Case Studies
Reedbush Supercomputer System

8••

University of TokyoI/O System by DDN►►

5 PB ExaScaler/Lustre @145.2 GB/sec
200TB of IME Burst Buffer
with NVMe @ 436.2
GB/sec© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

DDN.COM

DDN | EXAScaler & Lustre Case Studies
Kyoto University Supercomputer System9••

Kyoto University
I/O System by DDN►►

24PB ExaScaler/Lustre @150 GB/sec
230TB of IME Burst Buffer
with NVMe @ 240 GB/sec© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

DDN.COM

10

IMEシステム@Japanシステム

IME

ExaScaler(Lustre)

JCAHPC
Oakforest-PACS

物理960TB
25 x IME14K-OPA

物理32PB
10 x SFA14KXE-OPA

東大 Reedbush

物理230TB
6 x IME14K-EDR

物理6.25PB
3 x SFA14KE-EDR

京大 ACCMS2

物理230TB
6 x IME14K-OPA

物理24PB(P1: 16PB, P2: 8PB)
3 x SFA14K-EDR (P1)2 x SFA14KXE-EDR (P2)

合計

物理1.42PB

物理54.25PB (62.25PB)© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com11IME Performance, OPA, IOR on OFP

IOR 25 x IME14K (50 Nodes, 100RU)

GB/sec1TB/sec

1400
1200
10001204.31147.91184.2

1148.11028.7800
600
4002000

Posix Write
FPP

Posix Read
FPP

Posix WriteSSF

MPIIO WriteFPP

MPIIO Write
SSF

FPP : File Per Process
SSF : Single Shared File
© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com

12

FPP vs SSF►

FPP(File Per Process)• プロセス毎に独立したファイルを使用• メリット：並列ファイルシステムが得意、スケールしやすい• デメリット：超並列（何万・何十万プロセス）実行時など膨大なファイルによってメタデータ性能ネックになる►

SSF(Single Shared File)• 全てのプロセスで単一ファイルを共有• HDF5, NetCDFなどを利用• メリット：メタデータ性能ネックにならない• デメリット：FPPより場合によっては複雑© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.comLustre vs IME (IOR POSIX)
32 clients, 512 process, 3.3TB File Size

13IOR(Lustre, POSIX)
50000

50000

45000

45000

Hardware Limit

35000
30000
10K

25000

100K

20000

1000K

15000

35000
30000
10K

25000

100K

20000

1000K

15000

10000

10000

5000

5000

0Hardware Limit

40000
Throughput (MB/sec)

40000
Throughput (MB/sec)

IOR(IME, POSIX)

0WriteRead

FPP(File Per Process)

Write

Read

SSF(Single Shared File)FPP I/O Efficiency ~84%(Write) ~90%(Read)
SSP I/O Efficiency ~5%(Write) ~22%(Read)

Write

Read

FPP(File Per Process)

Write

Read

SSF(Single Shared File)FPP I/O Efficiency ~97%(Write) ~78%(Read)
SSP I/O Efficiency ~85%(Write) ~81%(Read)
Still under optimizations© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.comLustre vs IME (IOR MPI-IO)
32 clients, 512 process, 3.3TB File Size

14IOR(Lustre, MPIIO)
50000

50000

45000

45000

Hardware Limit

35000
30000
10K

25000

100K

20000

1000K

15000

35000
30000
10K

25000

100K

20000

1000K

15000

10000

10000

5000

5000

0

Hardware Limit

40000
Throughput (MB/sec)

40000
Throughput (MB/sec)

IOR(IME, MPIIO)

0
Write

Read

FPP(File Per Process)

Write

Read

SSF(Single Shared File)FPP I/O Efficiency ~84%(Write) ~90%(Read)
SSP I/O Efficiency ~5%(Write) ~22%(Read)

Write

Read

FPP(File Per Process)

Write

Read

SSF(Single Shared File)FPP I/O Efficiency ~97%(Write) ~78%(Read)
SSP I/O Efficiency ~97%(Write) ~81%(Read)
Still under optimizations© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com

15

WRF on IME
48 job, total 240 compute node – throughput – focus, 5 minute window

WRF output

WRF checkpointIME capacity consumption~390GB/s© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com16WRF on Lustre
48 job, total 240 compute node – throughput – focus, 5 minute window~100GB/s© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com17 WRF on IME at scale
Application runtime speedup over 1.5x

© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com18DDN | DDN IME Platforms►

IME120 (SuperMicro Platform)• Runs IME server software• 1P Broadwell, 1x EDR/OPA 1RU• 6 2.5” NVMe SSDs
• 64-128 GB DRAM @2400• 4.6-12TB IME Capacity• Max 10GB/s throughput per server►

IME240 (SuperMicro Platform)• Runs IME server software• 2P Broadwell, 2x EDR/OPA, 2 RU
• 20 2.5” NVMe SSDs
• 128 – 256 GB DRAM @2400• 9.6-40TB IME Capacity• Max 20 GB/s throughput per server►

IME14K Appliance• Two controllers in 4 RU
• 2P Haswell / Broadwell, EDR/OPA• 48 NVMe SSDs (V1.0.0) and 72 SAS SSDs (1H’16)• 38.4-86.4TB IME Capacity• 50 GB/s raw throughput per appliance

Note: Erasure coding topology will impact achievable peak performance!

© 2015 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.comThank You!
Keep in touch with us

sales@ddn.com

2929 Patrick Henry Drive
Santa Clara, CA 95054@ddn_limitless

1.800.837.2298
1.818.700.4000

company/datadirect-networks© 2014 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com