Lustre Performance Yesterday, Today and Tomorrow
Shuichi Ihara

Overview► Lustre continuously aims to improve performance, but it’s hard to confirm because..
• Many dependencies. (Not only Lustre version, but also kernel, OFED, Firmware version, etc.)• Need to understand test workload to see improvements• Same test workload, but it’s tested by different users or different cluster (hardware)• Not all performance improvements mentioned on the Lustre roadmap (need to read changelog)► Performance benchmark on same hardware with same configuration is important• Apples to apples testing with different Lustre branch• Tested three major Lustre versions (2.10.8, 2.12.2, and master)

We select and compare a number of interesting performance metrics on the different versions

whamcloud.com

Tested and Covered Workload► General Lustre Performance Comparisons on Flash System• Aggregate Throughput
o FPP (File Per Process), Strided SSF (Single Shared File) and Strided SSF Hardo 1MB IO Size on FPP and Strided and 47008 Byte IO size for Strided SSF Hard

• Aggregate IOPS (4K Random Read)• Single OSS Performance► Single Client Performance Comparisons• Single Thread• Single Client on High Network Bandwidth with Multi-rail• 4K Random Read IOPS► Metadata Operations for Small Files

whamcloud.com

Benchmark Configuration► Test Conditions• Keep same version on both server and client• Same Lustre parameters on each tests

Mellanox SB7700…

…

32 x FDR clients
CentOS7.6, OFED-4.5
168EDR16# lctl set_param obdfilter.*.brw_size=16# lctl set_param osc.*.max_rpcs_in_flight=16# lctl set_param osc.*.checksums=0# lctl llite.*.max_read_ahead_mb=2048• Enable new features if it improves performance
81x DDN AI400NV with 20 x 3.84TB SSD
Mixed 4 x OSS and MDS

whamcloud.com

General Lustre Performance Comparisons on Flash System

Confidential

Aggregate Throughput Comparison

FPP (File Per Process), Strided-SSF (Single Shared File), Strided-SSF-Hard
FPP:
ior -w/-r -s 6600 -i 1 -C -Q 1 -g -G 27 -k -e -t 1m -b 1m -F
Strided-SSF:
ior -w/-r -s 6600 -i 1 -C -Q 1 -g -G 27 -k -e -t 1m -b 1m
Strided-SSF-Hard: ior -w/-r -s 132000 -i 1 -C -Q 1 -g -G 27 -ke -t 47008 -b 47008
Write
60,000masterThroughput (MB/sec)

2.1x
2.9x

20,00010,00057x0

Almost Same
Performance

50,00040,00030,000b2_10

60,000b2_12

50,000Throughput (MB/sec)

Read

b2_10

59x

b2_12
master

40,00030,0002x

2.2xStrided-SSF

Strided-SSF-Hard

3x

20,00010,0000

FPP

Strided-SSF

Strided-SSF-Hard

FPP

whamcloud.com

Aggregate IOPS (4K Random Read, O_DIRECT)IOPS and B/W

IOPS

1600

► Workload is fio with random O_DIRECT readsB/W• Compare different IO sizes

1400

Network B/W Limit

IOPS (Kops/sec)

1200
1000

► 1536 processes on 32 clients with random reads► b2_12 and master 30% higher 4K IOPS vs. b2_10► b2_12 and master branch saturate network• 4x IB-EDR bandwidth is 48GB/sec at 64K IO size

800
600
400
2000

4K

8K

16K

32K

64K

128K

IO Size (Bytes)

b2_10

b2_12

masterwhamcloud.com

Single OSS Performance on Flash System
Heavy Concurrent Access N:1 (32 clients, 768 processes)
Single OSS Throughput

Single OSS Throughput (Time Chart)

15000+20%10,000

+20%
b2_10

7500

b2_12
master5000

Network Limit

5,000

05,000Consistent and stable
OST performance on
b2_12 and mastercompared to b2_10

Write0112233
4455
66
7788
99
110
121
132
143
154
165
176
187
198
209
220
231
242
253
264
275
286
297
308
319
330

10000

Network Bandwidth Limit

Throughput (MB/sec)

Throughput (MB/sec)

12500

15,000Read
b2_10

10,0002500

b2_12
master

15,000

0WriteReadwhamcloud.com

Lustre Client Performance ComparisonsConfidential

Single client IOPS Scalability(4K random read, O_DIRECT/AIO)
b2_10(NoAIO)

Single Client IOPS (4K Random Read)

b2_12

2000

• Compare different number of processes on one clientmaster1800► b2_10 doesn’t allow O_DIRECT+AIO► AIO works on b2_12 branch, but still “sync” mode
► master branch supports O_DIRECT+AIO► master delivers 1.7M IOPS 4K random read on
single client with sufficient storage performance

1600

IOPS (K ops/sec)

1400
1200

6.5x1000► “fio” generates random O_DIRECT+AIO reads

800
600
400
200
0050

100

150200250300

Number of Processwhamcloud.com

Single Thread Performance
Single Thread Performance
4000Throughput (MB/sec)

3500
3000

► 1MB single stream to a large file
# IOR -w (-r) -k -t 1m -b 384g -e -vv -o fileb2_10

► Flashed cache between write and read+82% ► Enabled Lustre file striping

b2_12
master+76%

2500# lfs setstripe -c -1 –S 16M► master 76-82% faster than 2.10/2.12 for read
► Write almost same speed on all branches

2000
1500
1000
500
0WriteReadwhamcloud.com

Single Client Performance with Multi-Rail► High Bandwidth Network (2 x IB-EDR) Client with Lustre Multi-Rail► File Per Process Workload (Aggregate 768GB files)► Hidden Client performance bottleneck behind Single IB-EDR bandwidth
Read

20,00020,00018,00018,00016,000+23%

14,0001x IB-EDR
12,000
B/W limit

+18%
+15%
+36%

10,000

b2_10

8,000

b2_12

6,000

master4,000

Throughput (MB/sec)

Throughput (MB/sec)

Write

16,00012,00010,000

b2_10

8,000

b2_12

6,000

master

4,000
2,000

001224

Number of Process

48+54%+37%

14,000

2,000
6+42%

6122448

Number of Processwhamcloud.com

Lustre Metadata Operations

Confidential

Metadata Performance
Small File with Non-Cached Operations
mdtest (Unique dir 32K files, Uncached Operation)
600,000
b2_10(NoCached)b2_12(NoCached)

500,000

master(NoCached)
400,000

b2_10(Cached)+60%

300,000► Workload is mdtest with create/stat/unlink• 11.5M x 32K files in unique directories► 32 clients 768 processes generate workload► No Cache aware mdtest# mdtest -n15000 -Fvu -w 32k -e 32k -N24 -d /path• Client calls stat()/unlink() on remote created files.
• Client cache at create is useless for next operations.

► Cache aware mdtest# mdtest -n15000 -Fvu -w 32k -e 32k -d /path• Opencache can be valid for stat/unlink• Measured on b2_10 branch as a reference numbers

200,000

100,000► master has 60% faster stat() than 2.10 and 2.12
► Unlink speed is almost same as cache aware mode

0FileCreation

FIleStat

FileRemoval

whamcloud.com

Single Client Small File Operation with DoM► Speed “untar“ operation
Time Comparision of "untar"
120%# tar xzf /lustre/linux-5.3-rc8.tgz -C /lustre/outb2_10b2_12
masterTime against b2_10

100%80%-10%-20%► Enabled DNE2 in all branch# lfs setdirstripe -c 4 /lustre/out# lfs setdirstripe -c 4 -D /lustre/out► DoM for less than 16M files# lctl set_param lod.*.dom_stripesize=16777216# lfs setstripe -L mdt -E 16M /lustre/out

60%► master branch improves untar operation by 10%► DoM can provide another 10% time reduction

40%
20%
0%

Non-DoM

DoM

whamcloud.com

Conclusions► Apples to Apples comparison (different branch on same hardware) is very helpful• Tested major three branches (2.10.8, 2.12.2, and master) on same hardware► master and b2_12 branch contains many performance improvements• Performance optimization for Flash system (both throughput and IOPS)• Client performance optimization (IOPS, bandwidth, reduced latency, remove overhead, etc.)• Metadata performance improvements for small files► Future work
• Add more performance metrics and different Lustre configurations (or hardware configurations. e.g. ethernet)• Found several performance bottlenecks in master that needs to be investigatedwhamcloud.com

Confidential

Backup: Tested Lustre Branches► b2_10 at v2_10_8• 1e117a3 New release 2.10.8► b2_12 at v2_12_2-150-g2e0e446bca• 2e0e446 Revert "LU-11816 lnet: setup health timeout defaults"► master at v2_12_57-22-g9b6b5e4798• 9b6b5e4 LU-11011 osc: add preferred checksum type support• Added the patches for LU-12518 and LU-4198

whamcloud.com