DDNの最新のLustreへの取り組みについて

DataDirect Networks, Inc.
2017/11/02

Shuichi Ihara© 2017 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com

2

DDN’s recent Lustre activities▶ DDN has been contributing Lustre community• DDN is No.2 company of Lustre cods contribution.
• DDN developed many new Lustre features and merged theminto upstream Lustre.
▶ Presenting at Lustre conference every year. Fourpresentations were selected at LAD17!

• Lustre on ARM servers• PCC(Lustre Persistent Client Cache) LU-10092• Lustre Audit with Changelogs LU-9727• Lustre/ldiskfs metadata performance boost LU-9796▶ Other activities• Lustre QoS (Corroboration work with University of Mainz)o Selected paper at SC17 (Tuesday, November 14th11:30am - 12pm)• Data Archive Solution• Lustre Integrated Policy Engine• Lustre-ZFS• ... others© 2017 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com3Lustre on ARM Servers© 2017 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.comThe Cavium ThunderX Architecture▶ SoC architecture• ISA: ARMV8root@s167:/proc# lscpu
Architecture:
aarch64
Byte Order:
LittleEndianCPU(s):96
On-line CPU(s) list:
0-95
Thread(s) per core:
1Core(s) per socket:
48
Socket(s):2
NUMA node(s):2
L1d cache:
32K
L1i cache:
78KL2
cache:
16384KNUMA node0 CPU(s):0-47
NUMA node1 CPU(s):48-95© 2017 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com

DDN Goals evaluating ARM▶ Understand if it is a viable option for mid/longterm future products▶ Understand what’s the effort necessary to
make Lustre running optimally on ARM (clientand server-side)▶ Understand how Lustre and general I/O
behaves on ARM SoC architecture▶ Contribute to the community© 2017 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.comTest Environment used for the study
ES7K IB
Embedded NL-SAS 7.2K

IB FDR (56Gbps)

s167

IB FDRIB FDRIB FDRIB FDRIB FDRIB FDR

s166

IB FDR 56 Gbps

s164

IB Switch 36p

s165

Es7k-vm1

Es7k-vm1
IB-SRP FDR (56Gbps)

IB FDR 56 Gbps

IB-SRP FDR (56Gbps)

SFA7700KX Block
Flash Array© 2017 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

s161

s162

s163

ddn.com

IOR Single Client Performance – MultipleThreads
IOR Single Client Performance - 4MB RPCs - REGULAR vs FAKE IO/mnt/arm/bin/ior.arm.mvapich -a POSIX -b 1g -r -w -F -B -t 4m -o /mnt/arm/file.out
6000

5000

4000

3000

2000

1000

012Writes 4M - Fake IO

4812Reads 4M - Fake IO

16Writes 4M - Real IO

243264

Reads 4M - Real IO© 2017 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.comARM and x86 Clients comparison
IOR, multiple clients - Sequential
ARM and x86 Clients - IOR Sequential Reads / Writes (ARM Server)
6000

5000

4000

3000

2000

1000

00

10
ARM Writes (16MB)

20

30
ARM Reads (16MB)

40x86 Writes (16MB)

50

60

70

x86 Reads (16MB)© 2017 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com

9PCC(Lustre Persistent Client Cache)© 2017 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.comNSCC-Wuxi and the Sunway Machine
FamilySunway-I:- CMA service, 1998
- commercial chip
- 0.384 Tflops- 48th of TOP500

Sunway BlueLight:- NSCC-Jinan, 2011
- 16-core processor- 1 Pflops- 14th of TOP500

Sunway TaihuLight:- NSCC-Wuxi, 2016
- 260-coreprocessor- 125 Pflops- 1st of TOP500

PCC project is collaborated by NSCC-Wuxi and DDN© 2017 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.comI/O Architecture of Sunway TaihuLight

Cache on I/O forwarding nodes (Lustre clients) should be helpful© 2017 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.comWhy SSD cache on Lustre client?
▶ Less overhead visible for applications• No network latency• No LDLM lock and other Lustre overhead▶ Easier to be optimized for the best performance• I/O stack is much simpler• No interference I/Os from other clients▶ Relatively easier than server side implementations• Write support for SSD cache on server side is very difficult• Problems for write cache on server side:o Visibility when failover happenso Consistency when corruption happens▶ Less requirement on hardwareo Any kind of SSD can be used as the cache device▶ Reduces the pressure of OSTso Small or random I/Os are regularized to big sequential I/Oso Temporary files do not need to be flushed to OSTs
© 2017 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.comArchitecture of PCC
Policy Engine
Control Flow
Data Flow
Cache Policy

PCC Switcher

Prefetch

Copytool #2

SSD
Copytool #3

Prefetch

OSCs

OST

OST

Cached I/O

Normal I/O

SSDOSCs

OST

PCC Switcher
Cached I/O

Normal I/O

SSD
Copytool #1

Cache Policy

PCC Switcher
Cached I/O

Normal I/O

Cache Policy

PrefetchOSCs

OSTOST© 2017 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.comLimitations▶ Not all applications are able to be accelerated by PCC• Locality requirements of application I/Oso Applications shall not access the cached file through multiple clientso But no inconsistency will happen even the application writes the cachedfile on a remote client• Capacity of each local cache is limitedo Size of a cached file is limited to the available space of the local cacheo The total cached data on a single client is limited▶ Files can not be partly cached• Partial cache can be implemented if HSM supports partialarchive/restore▶ The total PCC clients are limited to 32 Today• Only 32 different archive numbers are supported by Lustre• This upper limitation can be raised in the future© 2017 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com

15Lustre Audit with Changelogs© 2017 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.comNeed for audit in Lustre▶ Support of rich security features:• authentication with Kerberos• mandatory access control with SELinux• isolation• etc.
Þ Audit as a proof of security in place▶ Lustre outside of traditional HPC field▶ e.g. Life science• data privacy is crucialÞ Audit as a regulation compliance© 2017 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.comAudit with SELinuxPros• integrated
logging andauditing
facility• provenCons• on client side
• need toconsolidate© 2017 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.comAudit with Changelogs

Pros• integrated inLustre• centralized• transactionalCons• lacks someinfo© 2017 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.comAudit with Changelogs▶ Lustre activity as seen by MDS• file system namespace• file metadata▶ Store in Changelog records• internal Lustre files▶ Read from audit nodes• dedicated clients
5 01CREAT 15:44:32.385864793 2017.07.18 0x0 t=[0x200000402:0x3:0x0]• moveoutside for later analysisfileA
ef=0x1p=[0x200000402:0x2:0x0]© 2017 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.comLustre needs for proper audit▶ Identify subject of action• uid/gid• NID▶ Record all actions• open• close• xattr• denied accesses© 2017 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.comAudit with Changelogs: impact study▶ Changelogs space consumption evaluation# changelog
entrieschangelog
size

After 10 000
files created

30000

3755824

After 10 000
files read

50000

6096448▶AfterRule10of000
thumb: provision
B / entry on MDT
60000 125 7461440

files removed© 2017 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.comAudit with Changelogs: impact study© 2017 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com23

Lustre Metadata Performance
improvement© 2017 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com24

Why is metadata performance important?
▶ Lustre is general purpose filesystem for Big data

• 1 Million files per job are quite common with life science
application• AI/Machine learning type of workload requires small fileaccess with low latency. Metadata performance is one of key
factors of it.
• Lustre metadata performance has been performing well.
▶ Vertical and Horizontal scale• 28 (and 32) CPU cores/socket is available Today.
• DNE helps Horizontal scale out Metadata, but needs tounderstand your single MDS metadata performance first.

© 2017 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.comMDS-Survey
RHEL7.3/Lustre-2.10/ldiskfs

25

MDS-Survey(File Creation and Unlink)
RHEL7.3/Lustre-2.10.1RC/ldiskfs (Quota Enabled)
250,000

200,000

ops/sec

150,000
File Creation
File Removal

100,000

50,0000481632
NUmbers of Thread64128© 2017 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com26

A problem on File creation underconcurrency▶ Profiled with perf-tools during mdtest toldiskfs/ext4• Collected CPU costs for all functions in ext4 and jbd2
• Found heavy lock contentions on group spinlock
FUNC
ext4_create_raw_spin_lockjbd2__journal_start
jbd2_journal_get_write_accessext4_add_nondir
ext4_add_entry
ext4_dx_add_entry
ext4_mark_inode_dirty

TOTAL_TIME(us)1707443399
1317641501
287821030
33441470
29435963
26015166
25729337
12302433

COUNT
1440000
180899929
1453950
73077185
1440000
1440049
14328145774407

AVG(us)1185.72
7.28
197.96
0.46
20.44
18.07
17.96
2.13

• Same contentions exist in the upstream kernel© 2017 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com27▶

mdtest to ext4 (linux-4.13-rc5)

Fixed and merged upstream
kernel (4.14)

Default ext4

Wang Shilong (2):ext4: cleanup goto next groupext4: reduce lock contention in__ext4_new_inode

1200000

13x performance
improvement on file creation

800000• Run mdtest to ext4 directly• Unique directory operations• Quota disabled

patched ext4

1000000

ops/sec▶

Fix lock contentions in upstream kernel

13x
6000004000002000000File Creation

File removal© 2017 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.commds-survey on patched ldiskfs
LU-9796: speedup file creation under heavy concurrency▶ Ported patches to ldiskfs for RHEL7 kernel▶

File Creation :mds-survey on ldiskfs
1 x MDS and 1 x MDT(2 x RAID1 SSD)
300,000

250,000

200,000
ops/sec28

150,000

default
patch

100,000

50,0000481632
Number of Threads

64128© 2017 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com29Conclusion▶ DDN keeps investment to Lustre andcontributions to Lustre community• DDN Lustre R&D in Japan and China• Our most of developed new features comes from valuable
customer feedbacks!
▶ Deliver adaption and optimizations for new

hardware and new technology in advance• Performance lab is located in Tokyo• Various early testing, performance optimization areongoing▶ Welcome Co-research and Collaboration• Not only co-research, but also Alpha/Beta testing andfeedback are much appreciate!

© 2017 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change.

ddn.com