DDN Lustre Updates

DataDirect Networks Japan, Inc.
Shuichi Ihara
2015/10/22

DDN’s focused area at 2010-2014•Stability and Reliability– Lustre moved through Sun, Oracle, Whamcloud, Intel andOpenSFS– It was big transition pried of lustre-1.8 to lustre-2.x– Evolution of DDN storage system (FPGA based storage system
S2A9900 to SFA family)•Performance– Metadata performance– Maximize streaming performance, but minimize hardware

2

DDN Lustre Evolution
2010
Number of Lustre system in Japan

3

Capacity within two racks

1.2PB

Peak performance of Single LustreFS

250GB/sec

Performance/Rack

12GB/sec

Performance/Disk

50MB/sec

File creaOon speed/Single MDS

30,000 ops/sec

2015
16x +5x4x4x
2.5x
3.3x50+
6.7PB
1TB/sec
48GB/sec
125MB/sec
100,000 ops/sec

3

Lustre Overview//home/scratch/work

High Performance
NetworkMDS

...

MDS

OST n

OSS

OST n-1

OSS

OST 1

OSS

OST 0OSS

4IEEL: Intel Enhancements/

Hadoop Support/home/scratch/work
Stable Lustre Branch

OSS

OSS

MDS

OSS

MDS...

OST nIntel L3 support

OST n-1

OSS

OST 1

IML MonitoringHigh Performance support
Network

OST 0admin5ExScaler Performance Improvements//home/scratch/work

Lustre Client
mmap perf
Single thread perf

High Performance support
NetworkMDS

OSS

...

OST nOSSOST n-1

Throughput/Drive
32GB/s from 400 drives

OSS

OST 1OSS

OST 0admin

Bulk (16MB) RPC
Improved throughput

MDS

Metadata ServerOptimisation
200,000 file creates/s

SSD Acceleration
SFX support

6

ExaScaler Functional Improvements//home/scratch

Project Quotas
Flexible Quota provision/work

Replication
Parallel Data transfer

High Performance support
NetworkOSS

MDS

OSS...

MDSWorld’s largest Lustre
install base

OST nOSSOST n-1

OSS

OST 1

Fine Grained
Monitoring
Lustre Jobstatsadmin

OST 0QoS
Fine-grained ServiceManagement

High Density
Over 5PB per Rack

7

DDN EXAScaler/Hadoop Support/home/scratch

Project Quotas
Flexible Quota provision/work

Lustre Client
mmap perf
Single thread perf

SSD Acceleration
SFX support

OSS

OSS

MDS

OSS

MDS

Bulk (16MB) RPC
Improved throughput
World’s largest Lustre
install baseIntel L3 support...

OST nThroughput/Drive
32GB/s from 400 drives

OSS

OST n-1

Fine Grained
Monitoring
Lustre Jobstats

High Performance support
Network

OST 1IML Monitoringadmin

OST 0QoS
Fine-grained ServiceManagementReplication
Parallel Data transfer

Metadata ServerOptimisation
200,000 file creates/sHigh Density
Over 5PB per Rack8

Next DDN’s focused Area
We prioritized based on DDN Lustre feedbacks past 3 years.
• Performance– Small I/O and unlimited metadata performance– Speedup data migration and replication– Real time performance monitoring and analytics•

Corroboration, Security and Isolation– Project quota (Isolated space management)– Limited visible Filesystem’s Tree Structure (SubDir mount)– Trusted server and client9Design of L2RC(L2 Read Cache)•

Data flow

Currently focus on read-only Cache– Read support is relatively easy•

Control flow

All indexes are kept in memory– Worst-case: 1TB SSD will cost 10GB memory forindex•

Space allocation
–––––

•

All spaces are divided into 1M bucket
Bucket is the unit of space management
A bucket contains multiple extents
Extents size is between 4096 and 1M
Cached data of an object is divided to multiple 1M
extents and one variable-length extent

Prefetch and replace

ClientsCache hit readCache missreadPrefetch

Hot object list

– Automatic management based on file heat
– Driven by applications/users based on ladvise– Hooks are added in IO path to refresh cacheOSTs
10

Design of file heat•

Lustre file heat is a relative attribute which can reflect theaccess frequency of the file/object– It grows as file is accessed, yet dissipates as time moves on•

Time is divided into periods– The access number during each period is counted– The file heat is only recalculated at the end of a time period– At the end of each time period, a percentage of the former fileheat is lostLower heat
ColderLess access frequency

Higher heat
HotterMore access frequency

11What does file heat look like?
C: Access count per time period
P: Loss percentage per time period
450
400
350
300

C+11− p P

log 1

250

C/P

200
150
100
50
0
0

C=50

5

10

C=0

15

count

20

25

heat

30

C=200

3540

An idle period
12

L2RC with Lustre fadvise(Intentional Prefetch)lfs fadvise

Clients

ioctl API(enhancement of "lfs" comand)

Give advices

User space
Kernel space

LOV
Stripe to objectsOSC

OFD

OSC
Cache hit read

Cache hit read

OSS

OSS

SSD
Cache

LU_FADVISE_CACHE

HDD
OSD

Hints:
LU_FADVISE_NONE
LU_FADVISE_CACHE
LU_FADVISE_LOW
LU_FADVISE_MED
LU_FADVISE_HIGH

Prefetch

…

OFD

SSD
Cache

LU_FADVISE_CACHE

HDD
OSD

Prefetch

13

Benchmark results of a L2RC(Experimental)••

IOPS of multiple clients (IOR with 32 clients)
4 x SSD (440K theoretical IOPS)
4KB Random Read IOPS (HDD/SSD based OST vs OST/w L2RC)
500,000
450,000
400,000

IOPS (ops/sec)

350,000

10x300,000
250,000
200,000
150,000
100,000
50,0000
4K Random IOPS

40 x Spindle
Drive

80 x Spindle
Drive

160 x Spindle
Drive

160 x Spindle
Drive with L2RC

4 x OST on SSD

4 x SSD(RAID1)
Raw device

13,73926,06438,688389,232

416,994

440,428

14

Lustre Backup (Replication)

Two major phase
Speedup is necessary“Diff” detection(Filesystem Scanning)
Primary
LustreFilesystem

Copy data

Backup
LustreFilesystem

15

Limitations of current utilities•Traditional syncing tools (e.g. rsync, cp, ...)– Requires scanning of whole filesystems– Not designed for distributed filesystem– No Lustre changelog aware tool•

lustre_rsync (part of Lustre)– Support Lustre changelog and incremental backup, but...
– Still single thread copy tool
– Still based on rsync– Partial Changelog

16

Ldsync (Lustre dsync)
Parallel copytool based on Changelog•

Runs based on Lustre changelog– Analyze all sync required files from changlog– Clear old Lustre changelog after syncing if necessary•

Parallelization and scalability– Works with low level distributed sync tool
– Scaling copy operation (synchronization) by multi threads andmultiple nodes

17ldsync : Reduction of Scanning time
Scaning speed of 'Delta' in ﬁlesystems
1200

1000

TIme(sec)

800

600

ldsync
rsync

400

200

0
1.2

2.4

3.6

4.8

6

7.2

8.4

9.6

10.8

12

Milion ﬁle18

ldsync : Acceleration of Copy Speed

1TB Data ReplicaNon Time(MulNple Client)

1TB Data ReplicaNon Time(Single client)
600

2000
1800

500

1600

400

1200

TIme(sec)

TIme(sec)

1400

1000
800

300
200

600
400

100

200
0012418248

Number of Cleint

Number of Thread

Hardware
Limitation19DDN Storage Monitoring••

Flexible Data collector(Any statistics arepossible)
JOBID/GID/UID based Lustre monitoring•••

Lightweight
Near real-time
Massive Scale

BurstBuﬀer

Monitoring Servercollectd
Graphite plugin

LustreUDP(TCP)/IP
based small
text message
transfergraphite

GPFSStorageHardware

20

Thank you!

21