Community Release Update
October 2019
Peter Jones, Whamcloud
OpenSFS Lustre Working Group

OpenSFS Lustre Working Group
Lead by Peter Jones (Whamcloud) and Dustin Leverman (ORNL)
Single forum for all Lustre development matters• Oversees entire Lustre development cycle• Maintains the roadmap• Plans major releases• Collects requirements for future Lustre features• Sets priorities for test matrix
All are welcome to attend and/or join the mailing list
For more information visit the wiki
• http://wiki.opensfs.org/Lustre_Working_Group2

Lustre Community Survey (Mar 2019)
Which Lustre versions do you use in production? (select all that apply)
80.00%
70.00%
60.00%
50.00%
40.00%
30.00%
20.00%
10.00%
0.00%

Lustre 1.8.x Lustre 2.1.x Lustre 2.5.x

Lustre 2.6

Lustre 2.7.x Lustre 2.8.x

2018

Lustre 2.9

Lustre 2.10.x Lustre 2.11 Lustre 2.12.x

2019

Lustre 2.10.x LTS is the most widely-used production release3

Community Survey – Feature Usage
What describes your level of interest in...?
ZFS
Project Quotas
Progressive File Layouts
Multi-Rail LNETLFSCK (lctl)
IML
HSM
File Level Redundancy
DNE (Striped directories)DNE (remote directories)
Data on MDT
051015Will use within year

202530354045

Use in production

ZFS Production usage continues to grow (~22% 2016; ~36% 2018; 49% 2019)
DNE Striped Directories usage finally starting to get traction
Strong interest in using Project Quotas, PFL and Data on MDT

7Community Survey – Primary Usage
How would you characterize your primary usage of Lustre?
Other
Storage Vendor/Integrator
Particle Research
Meteorology
Media/Entertainment
Manufacturing/CAD/CAE
Government
Genomics/Life Sciences
Financial Services
EnergyDefense
Education
AI/Machine Learning
0.00%

5.00%10.00%

15.00%

20.00%

25.00%

30.00%

35.00%

40.00%

45.00%

50.00%• AI/ML, Life Sciences and Particle Research leading categories• Need to further refine categories based on feedback in Other

8

Lustre Kernel Policy• Confusion for a number of years about Linux distro support• Questions about omissions from the Lustre support matrix• “Support” is an overloaded term
• Many more distros than listed work, but these are the ones tested• Should developers provide fixes for all kernels in the tree?

• LWG discussed and decided to clarify• Developers only need to fix primary distros when making changes• Other distros rely on interested parties maintaining them
• Lustre/Changelog updated to make clearer distinction• Policy on Lustre.org wiki

6

Lustre LTS Transition• LTS change from 2.10.x to 2.12.x announced at SC18
• Driver was amount of change to support newer kernels• Some concerns have been raised that two years is not LT J
• Large numbers of sites deployed 2.10.x LTS releases• Some sites upgraded fully to 2.12.x• Some sites using 2.12.x clients• Some sites work with vendors who offer longer support versions• Some sites using changes ported back to 2.10.x if required• Will again seek community input before any future changes

7

Lustre 2.12.3• Lustre 2.12.3 RC1 is undergoing release testing• RHEL 7.7 server and client support• RHEL 8 client support• MOFED 4.7• ZFS 0.7.13• Bug fixes from early 2.12.x deployments• http://wiki.lustre.org/Lustre_2.12.3_Changelog

8

Lustre 2.13
• Trending for Q4 2019 release• OS support• RHEL 7.7 servers/clients• RHEL8/SLES12 SP4/Ubuntu 18.04 clients• Interop/upgrades from latest Lustre 2.12.x• Ships with ZFS 0.7.13 by default• Number of useful features• Persistent Client Cache (LU-10092)• Overstriping (LU-9846)• Self Extending Layouts (LU-10070)• http://wiki.lustre.org/Release_2.13.0

9

Lustre 2.13 Contributions
HPE12

LLNL
ANL 81 ANU 30
19Intel 2926
Amazon 264
Uber 122
GSI Inbase
42
NASA294
21
Seagate620
Atos 20
Fujitsu
CEA 29ORNL
SUSE 3020
IU896
Other3266
335382
DUG 8

Ali Baba 3
Stanford 2Intel 7 LLNL 10 SUSE 25
Other 20IU 2
HPE 50
GSI 3

Cray 10678

DESY 1

Uber 6

Amazon 3
CEA 6

ORNL 175
Cray 408

LINES OF CODE

PATCH REVIEWS

DDN-Whamcloud 110577

DDN-Whamcloud 1767

Source: http://git.whamcloud.com/fs/lustre-release.git/shortlog/refs/heads/master

Aggregated data by organization between 2.12.50 and 2.12.58 tags

Data courtesy of Dustin Leverman (ORNL)Lustre Version Statistics
VersionCommits

LOC

Developers

Organizations

1.8.0

997

291K

41

1

2.1.0

752

92K

55

72.2.0

329

58K

42

102.3.0

586

87K

52

132.4.0

1123

348K

69192.5.0

471

102K

70

15

2.6.0

885

147K

76

14

2.7.0

742

201K

65

152.8.0

995

147K

92

172.9.0

737

74K

121

162.10.0

732

108K

85

142.11.0

860

134K

87

18

2.12.0

697

82K

90

19

2.13.0 (to 2.12.58)

853

138K

65

24

Statistics courtesy of Chris Morrone (LLNL)/ Dustin Leverman (ORNL)

Source: http://git.whamcloud.com/fs/lustre-release.git

11Lustre Community Roadmap
2018
Q4

Q1

2019

Q2

Q3

Q4

Q1

2020

Q2

Q3

Q4

2021
Q1

2.12.0FeatureReleases

2.13
2.14
2.15

LTS
Updates

2.10.62.10.7

2.12.X LTS 2.12.1
Release Stream

LEGEND:

2.12• Lazy Size on MDT• LNet Health• DNE Dir Restriping

2.10.82.10.X LTS
Release Stream
2.12.32.12.2
Expected
Timeline

2.13• Persistent Client Cache• Multi-Rail Routing• Overstriping

LTS releases
continue…..
Timeline
TBD

Completed

2.14• FLR Erasure Coding• Pool Quota• DNE Auto Restriping* Estimates are not commitments and are provided for informational purposes only
* Fuller details of features in development are available at http://wiki.lustre.org/Projects

LTS Branch

2.15• Client Encryption• Writeback CacheLustre in Linux Kernel• SUSE/ORNL working on getting Lustre client into Linux kernel• https://github.com/neilbrown/lustre• Patches pushed for review on lustre-devel mailing list

• Major ldiskfs patches merged into upstream ext4/e2fsprogs• Now much easier to keep Lustre e2fsprogs current

Lustre in the Cloud• Intel launched AWS offering early 2015
• Cray Azure offering has been available since late 2017
• Amazon announced FSX for Lustre Software in Nov 2018
• DDN-Whamcloud have offerings on GCP, Azure and AWS• Amazon and Ali Cloud both contributed to Lustre 2.13

Integrated Manager for Lustre
https://github.com/whamcloud/integrated-manager-for-lustre/releases• IML 5.0 GA in May• Compatible with Lustre 2.12.x• Lower manager/agent resource usage• IML 5.1 due out any day• https://github.com/whamcloud/integratedmanager-for-lustre/releases/tag/v5.1.015

Lustre Release Documentation• Latest version of manual dynamically available to download• http://lustre.org/documentation/• Also links for how to contribute• If you know of issues then please open an LUDOC ticket• If you do not have time to submit a patch then unformatted text
provides a starting point for someone else to complete• Large amount of content exists on lustre.org• http://wiki.lustre.org/Category:Lustre_Systems_Administration• Lustre Internals content being refreshed16

Lustre’s Longevity• Project active for over 20 years• Maintaining stability andperformance in the mostdemanding environments is notan easy problem to solve• Constant evolution to deal withchanging requirements ofhardware and usage• Permissive open source license
has meant many organizations
can use and contribute effectively• For some further Lustre heritage…
https://hps.vi4io.org/_media/events/2019/
hpc-iodc-lustre_next_20_years-dilger.pdf

Summary• LTS model has been well adopted§ Focus currently on switching to 2.12.x LTS• Lustre 2.13 is coming soon
• Lustre 2.12.3 is coming sooner• IML 5.1 coming out even sooner J• Plenty of options for those interested in contributing to Lustre• LWG http://wiki.opensfs.org/Lustre_Working_Group24

Thank you

Open Scalable File Systems, Inc.
3855 SW 153rd DriveBeaverton, OR 97006
Ph: 503-619-0561
Fax: 503-644-6708
admin@opensfs.org

www.opensfs.org