2014/10/17

DDN Japan Lustre Updates

Robert Triendl, DDN Japan©2012 DataDirect Networks. All Rights Reserved.

1ddn.com

HPC Storage for next a couple years

Computeand Data

FastData

Persistent
Data

Distributed
Data

INTELLIGENT POLICY & RESOURCEMANAGEMENT©2012 DataDirect Networks. All Rights Reserved.

2ddn.comLustre Markets
Life Sciences
Data Analysis Academia

Particle Physics Data Analysis
Astronomy Data Analysis
Genome Sequencing

Lustre Data
Data Analysis Government
Commercial Data Analysis

Image Analysis
Other Security
Shared-Memory/In-Memory
Hadoop on Lustre
Physics/Materials Science/ChemistryApplication Centers
Lustre Work

Particle Physics
Weather & Climate
Earth Sciences

Multi-Application/Academic Centers
Lustre as Tier 2 and Disk Archive

AcademicCorporate
Lustre Tier 2
On-Premise Tape

Archive

Archive Solutions for Lustre

On-Premise Disk
Replicated ArchiveCloud©2012 DataDirect Networks. All Rights Reserved.

3ddn.comRAS Features	
Management	
Connectors	
NFS/CIFS Access	

Security Features	IME Caching	

HPC Work
Corporate, 12%

Business Data
Analysis, 4%

Lustre WAN	

Government/
Security Data
Analysis, 17%

Large I/O	

Project Quota	
Client Performance	

HPC Work, 20%

Small File I/O	Cluster Integration	
Data Management	

Research Data
Analysis, 28%

Metadata Perf	HPC Archive,18%

SSD Acceleration	

Backup/Replication	
Object/Cloud Links	

©2012 DataDirect
Networks.
All Rights
Reserved.
©2012
DataDirect
Networks.
All Rights
Reserved.

HSM	

Fine-Grained Monitoring	

4ddn.com
ddn.comUnified Hardware Platform▶ Build around NVMe
▶ Intel Haswell/Broadwell

processor▶ Integrated High IOPs andIME SSD Box▶ Direct Connection for 10

Front/Side Isometric View

8460 Enclosures

Right Side View
©2012 DataDirect Networks. All Rights Reserved.

5ddn.com

SFA14KE, Codename “WolfCreek-E”
A Versatile, Integrated IOPS and Bandwidth Centric Storage Platform300 GB/sec	60 GB/sec	

30 GB/sec	

40U rack-optimized
configuration,including networking
Embedded OSSs
Up to 300 GB/secper Rack with SSDs
Up to 4 PB per Rack

SFA14K/14KE
••
•
•••
•

Compact 4U packaging
40-60GB/s Sustained Bandwidth30-40GB/s Embedded Option
Supports SAS 2.5” and PCIe
Connect To 10 x SS8460 JBODs
Controller-Resident SFX Cache
Support for Controller Clustering
6©2012 DataDirect Networks. All Rights Reserved.

Scalable MDS Unitsand ManagementServerDDN Confidential – NDA Required, Roadmap Subject to Change 6ddn.com

EXT4 Classification Details▶ OS changes (block layer)• Add classifier to I/O requests• Only coalesce like-class requests• Copy classifier into SCSI CDB▶ ext4 changes• 18 classes identified• Optimized for file server▶ Small kernel patch▶ One-time change to the FS©2012 DataDirect Networks. All Rights Reserved.

Ext3ClassGroup
NumberCache
priority

Unclassified

012

Superblock10Group desc.

20

Bitmap30

Inode40

Indirect block

50Directories

60Journal

70

File <= 4KB

81File <= 16KB

92File <= 64KB

103…

…

…

File > 1GB18

117ddn.com

FileUtils Project (http://fileutils.io)©2012 DataDirect Networks. All Rights Reserved.

8ddn.com

HPC Storage for next a couple years

Computeand Data

FastData

Persistent
Data

Distributed
Data

INTELLIGENT POLICY & RESOURCEMANAGEMENT©2012 DataDirect Networks. All Rights Reserved.

9ddn.comWOS-Bridge: GRIDScaler and EXAScaler
Collaborate, Distribute, Federate
Site 1
ComputeGrid

Site 3

WOS-Bridge,
Storage Cloud

Remote
CIFS/NFS
Users

EXAScaler(DDN Lustrecluster)

WOS-Access

Site 2Site 4ComputeGridPublicCloudGRIDScaler(DDN GPFScluster)

10©2012 DataDirect Networks. All Rights Reserved.

S3

DDN10
NDA CONFIDENTIAL

ddn.com

DDN APAC Lustre Development andSupport Infrastructure
Lustre Community Bug tracking
system	A repository of Lustre
mainstream 	

merge
charrypickup	

Lustre support tracking
system for Japan and Asiancustomer	

http://
redmine.datadirectnet.jp	

After gatekeeper
approved, codes are
merged into git tree	

Code review process
for merge them into
mainstream	

http://autotest.datadirectnet.jp	
Standard Lustre regression
tests
and DDN Specifc stress tests	

http://git.datadirectnet.jp	
Build new Lustre RPMs
from committed codes 	

http://jenkins.datadirectnet.jp	

Commit codes 	SupportEscalation from the customers	Development
Project management and tracking	* Password is required for all DDN Infrastructure access.	
©2012 DataDirect Networks. All Rights Reserved.

11

ddn.com10/17/2014

DDN Lustre Updates
DataDirectNetworks Japan, Inc
Shuichi Ihara©2012 DataDirect Networks. All Rights Reserved.

12

ddn.com

DDN Japan's Lustre Investigation▶ Corroboration and contributions to community• Number #2 (#3 LOC level) Lustre contributed vendor for Lustre-2.6• Focused on usability and manageabilityo Lustre Performance Monitoringo Project Quota(will support project quota in Ext4 too!)o filutils (LLNL, ORNL, LLAL, etc)

▶ Performance, (Re-)benchmark, analysis and optimization• Fundamental benchmark on new hardware• Optimize to DDN hardware• Integration with SSD devices and transparent data access▶ Lustre with IME (DDN Burst Buffer Technology)©2012 DataDirect Networks. All Rights Reserved.

13

ddn.com

JOB aware I/O performance monitoring▶ Developed Lustre Performance monitoring Tool

• Near real time data point collection. (every second)• Any type of I/O monitoring is possible.
(UID/GID/JOBID or any type of custom ID)
ExaScaler Monitor	

Total	

User1	

User2	

Group1	14©2012 DataDirect Networks. All Rights Reserved.

Group2	

14ddn.com

Recent focused interesting Lustre
performance improvements and testing▶ Large bulk I/O size support and enhancement▶ mmap() I/O performance improvements▶ Performance improvements for single shared file▶ CPU/Memory resource reduction▶ Fundamental metadata performance benchmark

15©2012 DataDirect Networks. All Rights Reserved.

15

ddn.comLarge bulk I/O support and enhancement

16| Engineering TechnicalConference 2014©2012 DataDirect Networks. All Rights Reserved.

Lustre on SFA12KX(320 x 4TB NL-SAS)	
35

30

25GB/sec	# cat /proc/fs/lustre/obdfilter/*/brw_stats!
snapshot_time:
1406696961.271996 (secs.usecs)!
read|write!
pages per bulk r/w
rpcs % cum % | rpcs
% cum %!
1:1091416
11| 681741
22!
2:
62166
01| 164562
02!
4:
96568
01| 60799
02!
8:11594501| 1005402!
16:170813
01| 11361
02!
32:242152
01| 18944
02!
64:44482702| 37609
02!
128:861561
03| 107677
03!
256:
99436837 96 100| 32549912 96 100!
!
read|write!
discontiguous pages
rpcs % cum % | rpcs
% cum %!
0:102060933 99 99| 33641331 99 99!
1:1778500 99| 1196
0 99!
2:
27307
0 99
|
39
0 99!
3:
10447
0 99
|
27
0 99!
4:
5502
0 99
|
16
0 99!
!
- snip – !
!
read|write!
discontiguous blocks
rpcs % cum % | rpcs
% cum %!
0:102029460 99 99| 31615681 93 93!
1:2088940 99| 2026762
6 99!
2:
27592
0 99
| 1310 99!
3:105110 99|
250 99!
4:55490 99|
90 99!
!
- snip -!

20WriteRead151050
1MB I/O

4MB I/O

1616MB I/O

ddn.comLustre Performance Degradation withLarge Number of Threads
IOR (FFP, 1MB) to single OST▶ 20 clients and up to 640 processes
Write: Lustre Backend Performance Degradation(Maximum for each dataset=100%)

Read : Lustre Backend Performance Degradation(Maximum for each dataset=100%)

7.2KSAS(1MB RPC)

7.2KSAS(1MB RPC)

7.2KSAS(4MB RPC)

SSD(1MB RPC)

120%

Performance per OST as % of Peak Performance 	Performance per OST as % of Peak Performance 	

▶

100%

80%60%

40%20%0%

SSD(1MB RPC)

120%

100%80%60%

40%20%0%0100

200

300

400

500

Number of Process	

17	7.2KSAS(4MB RPC)

©2012 DataDirect Networks. All Rights Reserved.

600

700

0100200

300

400

500

600

700

Number of Process	

17ddn.com

mmap() I/O performance improvements
Several application calls a lot of mmap().10%+of open() calls with mmap()!	

mmap() read perforamnce(1MB block size)
500
400# cat /proc/fs/lustre/llite/*/stats!
llite.share1-ffff881067f9b800.stats=!
snapshot_time
1408263676.546716 secs.usecs!
read_bytes
589388 samples [bytes] 0 2147479552 258867698600!
write_bytes
1025093126 samples [bytes] 1 4194304 637173439272!
osc_read3880442 samples [bytes] 8 1048576 3667025741928!
osc_write
640640 samples [bytes] 5 1048576 637252863026!
ioctl
17938 samples [regs]!
open
90267 samples [regs]!
close90239 samples [regs]!
mmap
10523 samples [regs]!
seek
6997546 samples [regs]!
fsync
16 samples [regs]!
readdir
48874 samples [regs]!
setattr
252 samples [regs]!
truncate
12 samples [regs]!
getattr
2097773 samples [regs]!
create3465 samples [regs]!
link
1 samples [regs]!
unlink2890 samples [regs]!
statfs2069 samples [regs]!
alloc_inode8423 samples [regs]!
getxattr
1025105141 samples [regs]!
inode_permission
229899278 samples [regs]	

300
200
1000
lustre-1.8.9

lustre-2.5.2

DDN branchmmap() read perforamnce improvements	
Lustre-1.8.9

DDN branch

128K 18512K

500
400
300
200
100018©2012 DataDirect Networks. All Rights Reserved.

32K1024K
ddn.comLustre Metadata Performance Impact
MDS's CPU speed▶

Metadata Performance comparison (Unique Directory)• 32 clients, 1024 processes, 1.28M Files• Tested on 16 CPU cores with 2.1, 2.5, 2.8, 3.3 and 3.6GHz CPU Speed
Directory Operation(Unique)	
Dir Creation

Dir Stats

File Operation(Unique)	

Dir Removal

File Creation

180%

180%

160%

160%

140%

140%

120%

120%

38%	

20%	

100%

100%

80%

80%

60%

60%

40%

40%

20%

20%

0%
2.1GHz

2.5GHz2.8GHz3.3GHz©2012 DataDirect Networks. All Rights Reserved.

3.6GHzFie Stats

File Removal

60%	
38%	

0%70%	

2.1GHz

2.5GHz2.8GHz193.3GHz

3.6GHzddn.comLustre Metadata Performance Impact
MDS's CPU speed▶

Metadata Performance comparison (Shared Directory)• 32 clients, 1024 processes, 1.28M Files• Tested on 16 CPU cores with 2.1, 2.5, 2.8, 3.3 and 3.6GHz CPU Speed
Directory Operation(Shared)	
Dir Creation

Dir Stats

File Operation(Shared)	

Dir Removal

File Creation

180%

180%

160%

160%

140%

140%

50%	

120%

120%

100%

100%

80%

80%

60%

60%

40%

40%

20%

20%

22%	

0%
2.1GHz

2.5GHz2.8GHz3.3GHz©2012 DataDirect Networks. All Rights Reserved.

3.6GHzFie Stats

File Removal

58%	
30%	0%
2.1GHz

2.5GHz2.8GHz20

3.3GHz

3.6GHzddn.comLustre Metadata Performance Impact
MDS's CPU Cores▶

Metadata Performance comparison (Unique Directory)• 32 clients, 1024 processes, 1.28M Files• Tested on 3.3GHz CPU speed with 8, 12 and 16 CPU cores w/wo logical processors
Directory Operation(Unique)	
Dir Creation

Dir Stats

File Operation(Unique)	

Dir Removal

File Creation

250%

250%

200%

200%150%150%

Fie Stats

File Removal

180-220%	

25%	100%

100%

50%

50%

0%
8CPU

8CPU_HT

12CPU 12CPU_HT 16CPU 16CPU_HT©2012 DataDirect Networks. All Rights Reserved.

0%50%	

8CPU

8CPU_HT

12CPU 12CPU_HT 16CPU 16CPU_HT

21ddn.com

Lustre Metadata Performance Impact
MDS's CPU speed▶

Metadata Performance comparison (Unique Directory)• 32 clients, 1024 processes, 1.28M Files• Tested on 3.3GHz CPU speed with 8, 12 and 16 CPU cores w/wo logical processors
Directory Operation(Shared)	
Dir Creation

Dir Stats

File Operation(Shared)	

Dir Removal

File Creation

160%

180%

140%160%

Fie Stats

File Removal

Creation(Sshare) andStat do not scale	

140%

120%

120%

100%

100%
80%
80%
60%
60%
40%

40%

20%

20%0%
8CPU

8CPU_HT

12CPU 12CPU_HT 16CPU 16CPU_HT©2012 DataDirect Networks. All Rights Reserved.

0%8CPU

8CPU_HT

12CPU 12CPU_HT 16CPU 16CPU_HT

22ddn.com

Lustre Metadata Performance
MDSs Scalability (Unique Directory)
Lustre Metadata Scalability (Unique)(32 clients, 1024 mount points)	
250000200000150000
ops/sec	File Creation
File Removal
Dir Creation
Dir Removal

100000File Creation(DNE)
File Removal(DNE)
50000

016

3264128

2565121024

Number of Threads	

©2012 DataDirect Networks. All Rights Reserved.

23ddn.com

HPC Storage for next a couple years

Computeand Data

FastDataPersistent
Data

Distributed
Data

INTELLIGENT POLICY & RESOURCEMANAGEMENT©2012 DataDirect Networks. All Rights Reserved.

24

ddn.comIME Demo at ISC 2014
▶ 19 Client Nodes• Commodity x86 servers
8 IME Servers~80 GB/sec	

18 IME Clients
MPI/IO

Parallel FS~3-4 GB/sec©2012 DataDirect Networks. All Rights Reserved.

▶ 8 IME Servers• Commodity x86 servers
with two IB FDR links• 196 SATA SSDs
• Two Separate IME Pools▶ Parallel FS• DDN GridScaler• 60 NL-SAS drives• 2 NSD servers• 3-4 GB/sec write and readthroughput

25ddn.com

IME Demo at ISC 2014
IOR Write(MPI-IO)	

Application
Write I/O	IOR Read(MPI-IO)	

©2012 DataDirect Networks. All Rights Reserved.

26

ddn.com