JLUG2019

Topics from LUG 2019, ISC19, LAD& Lustre Developer Summit 2019
Shinji Sumimoto, Ph.D.
FUJITSU LIMITED Oct. 17th, 2019

Copyright 2019 FUJITSU LIMITEDOutline of This Talk
Lustre 20 year’s Anniversary
 ISC 19 workshop slides from Andreas

Fugaku Update and FEFS
(Lustre based File System) Issues and Directions
Topics from LUG19 and LAD19
 LNET Multi-rail Improvement

1

Copyright 2019 FUJITSU LIMITEDLustre 20 year’s Anniversary

2

Copyright 2019 FUJITSU LIMITEDLustre History
 1999: Lustre file system 研究プロジェクト開始 by Peter J. Braam 2001: Cluster File Systems社設立
 2007/9: Sun社がCluster File Systems社を買収
 2009/4： Oracle社がSun社を買収、コミュニティに不安が走る
 2010/4: Oracle社がサポートを自社ハードに制限、コミュニティ激震
 欧州EOFS、米国国研ベースOpenSFS、World WideユーザベースHPCFSの３つのコミュニティに分散 2010/9: Whamcloud社設立
 Oracle社からWhamcloud社に技術者が集結

 2010/12: Oracle社がLustre開発凍結
 2011/4: LUG2011で３つのコミュニティがOpen SFS+EOFSとして再始動
 2012/7: Intel社がWhamcloud社を買収
 2018/6: DDN社がIntelのLustre事業を買収、新生Whamcloud誕生

2019: Luster 20 years anniversary
3

Copyright 20192011 FUJITSU LIMITEDLUG2009: Lustre 10th Anniversary

4Copyright 2019 FUJITSU LIMITEDLUG 2011: Single Community

5

Copyright 2019 FUJITSU LIMITED2018/6: DDN社がIntelのLustre事業を買収

6

Copyright 2019 FUJITSU LIMITEDFrom ISC19 Workshop Slides by Andreas
https://hps.vi4io.org/_media/events/2019/hpciodc-lustre_next_20_years-dilger.pdf

7

Copyright 2019 FUJITSU LIMITEDFrom ISC19 Workshop: Lustre – The Next 20 Years
https://hps.vi4io.org/_media/events/2019/hpciodc-lustre_next_20_years-dilger.pdf

8

Copyright 2019 FUJITSU LIMITEDFrom ISC19 Workshop: Lustre – The Next 20 Years
https://hps.vi4io.org/_media/events/2019/hpciodc-lustre_next_20_years-dilger.pdf

Young Andreas!?

9

Copyright 2019 FUJITSU LIMITEDFrom ISC19 Workshop: Lustre – The Next 20 Years
https://hps.vi4io.org/_media/events/2019/hpciodc-lustre_next_20_years-dilger.pdf

10

Copyright 2019 FUJITSU LIMITEDFrom ISC19 Workshop: Lustre – The Next 20 Years
https://hps.vi4io.org/_media/events/2019/hpciodc-lustre_next_20_years-dilger.pdf

11

Copyright 2019 FUJITSU LIMITEDFrom ISC19 Workshop: Lustre – The Next 20 Years
https://hps.vi4io.org/_media/events/2019/hpciodc-lustre_next_20_years-dilger.pdf

12Copyright 2019 FUJITSU LIMITEDFrom ISC19 Workshop: Lustre – The Next 20 Years
https://hps.vi4io.org/_media/events/2019/hpciodc-lustre_next_20_years-dilger.pdf

13

Copyright 2019 FUJITSU LIMITEDFrom ISC19 Workshop: Lustre – The Next 20 Years
https://hps.vi4io.org/_media/events/2019/hpciodc-lustre_next_20_years-dilger.pdf

14

Copyright 2019 FUJITSU LIMITEDLustre Roadmap@LUG2009 by Peter Bojanic
http://wiki.lustre.org/images/3/31/PeterBojanic.pdf

15

Copyright 2019 FUJITSU LIMITEDFrom "BlueGene/L Applications, Algorithms and Architectures"Workshop, August 13–14, 2002, Lake Tahoe, CA https://asc.llnl.gov/computing_resources/bluegenel/talks/braam.pdf

16

Copyright 2019 FUJITSU LIMITEDFrom "BlueGene/L Applications, Algorithms and Architectures"
Workshop: Lustre The Inter-Galactic File System https://asc.llnl.gov/computing_resources/
bluegenel/talks/braam.pdf

17

Copyright 2019 FUJITSU LIMITEDFrom "BlueGene/L Applications, Algorithms and Architectures"
Workshop: Lustre The Inter-Galactic File System https://asc.llnl.gov/computing_resources/bluegenel/talks/braam.pdf

18

Copyright 2019 FUJITSU LIMITEDFrom "BlueGene/L Applications, Algorithms and Architectures"
Workshop: Lustre The Inter-Galactic File System

19

Copyright 2019 FUJITSU LIMITEDFrom "BlueGene/L Applications, Algorithms and Architectures"
Workshop: Lustre The Inter-Galactic File System

20

Copyright 2019 FUJITSU LIMITEDFrom "BlueGene/L Applications, Algorithms and Architectures"
Workshop: Lustre The Inter-Galactic File System

21

Copyright 2019 FUJITSU LIMITEDFrom "BlueGene/L Applications, Algorithms and Architectures"
Workshop: Lustre The Inter-Galactic File System

22

Copyright 2019 FUJITSU LIMITEDFugaku Update and FEFS (Lustre based FileSystem) Issues and Directions23

Copyright 2019 FUJITSU LIMITEDFrom ISC19 Post-K Related Presentation byProf. Ishikawa

24

Copyright 2019 FUJITSU LIMITEDUpdate on Fugaku Development(1)25Copyright 2019 FUJITSU LIMITEDUpdate on Fugaku Development(2)26Copyright 2019 FUJITSU LIMITEDFugaku Software Stack (3)27Copyright 2019 FUJITSU LIMITEDUpdate on Fugaku Development(4)28Copyright 2019 FUJITSU LIMITED

1 Peta FLOPS System: K computer vs. Post-K
 Post-K (Now Fugaku)

 K computer 1x rack w/ SSDs

SPARC64 VIIIfx+ ICC

K computer

Post-K

Compute nodes

7,680(=96x80)

IO nodes4,80(=6x80)

Footprint (m2)128(=4x32)

1.1

SPARC LinuxArm Linux

384

More applications as well as
system software will come incollaboration withOpen Source Community

29

A64FX2.2m

 80x compute racks & 20x disk racks

Copyright 2019 FUJITSU LIMITED次世代共有ファイルシステムの課題 (JLUG2016)
 ファイルアクセス性能向上
 メタデータアクセス性能向上
 単体I/O処理高速化

 耐故障性向上
 フェイルオーバ時のI/O エラー防止 ファイルサーバ異常発生からファイルI/O 再開までの時間短縮
 両系故障時のアクセスハング抑止 保守性向上 ファイルシステム復旧時間短縮

 負荷耐性向上：メタアクセス、データアクセス
 MDS高負荷防止: 大量ファイルアクセス時のメタアクセスレスポンス低下防止
 インタコネクト高負荷時のノード異常の誤検出回避 アクセス公平性確保
 特定プロセスのファイルアクセスによりほかのプロセスが大きく影響を受ける

 省メモリ性向上30

Copyright 2019 FUJITSU LIMITEDThe 1st R-CCS International Symposium 18 Feb,2019"Operation of the K computer and the facility“ by Dr. Shoji

https://www.r-ccs.riken.jp/R-CCS-Symposium/2019/slides/shoji.pdf

31

Copyright 2019 FUJITSU LIMITEDSlide from "Operation of the K computer and thefacility“ by Dr. Shoji
https://www.r-ccs.riken.jp/R-CCS-Symposium/2019/slides/shoji.pdf

32

Copyright 2019 FUJITSU LIMITEDSlide from "Operation of the K computer and thefacility“ by Dr. Shoji
https://www.r-ccs.riken.jp/R-CCS-Symposium/2019/slides/shoji.pdf

33

Copyright 2019 FUJITSU LIMITEDFugakuにおける課題解決のアプローチ
 階層ファイルシステムの導入
 SSD採用の第１階層LLIO、第２階層のLustreベースファイルシステム
 16ノード毎にLLIOサーバSIOを配置、第２階層はSIO毎にマウント

 SSD採用の第１階層LLIO導入効果
 アプリケーションファイルアクセスの高速化•ノードローカル＆ジョブ内共有＆第２階層キャッシュアクセス(SSD base) 第２階層のLustreベースファイルシステムへの負荷削減と制御性向上•クライアント数、京: 10万規模→Fugaku: 1万規模
•第２階層サーバのメモリ使用量の削減、ネットワーク負荷の削減

 SSDはジョブ実行中の一時データのみ格納： SSDは非冗長構成•高速性確保と機器コスト削減•SSD, SIO故障：• 関連ジョブ異常終了で処置： 故障部品交換でリカバリ処理不要• 影響は小数Jobに限定： 負荷分散とシステム全体の故障耐性向上

34

Copyright 2019 FUJITSU LIMITED第２階層のLustreベースファイルシステム:FEFS Lustre 2.x ベースで開発・試験開始： 状況に応じてUpdate コミュニティコードベース開発が基本： 将来のUpdate障壁緩和

 耐故障性向上
 サーバ異常発生からファイルI/O 再開までの時間短縮：Lnet MultiRail改善 保守性向上 ファイルシステム復旧時間短縮: ストレージのデバイスリカバリ高速化等

 負荷耐性向上
 MDS高負荷防止: DNE, DNE2採用によるメタデータ処理分散

 アクセス公平性確保
 QoS機能の導入

35

Copyright 2019 FUJITSU LIMITEDFugaku開発: Lustreコミュニティ連携
 2019/8 京運用停止、2020年Fugakuインストール本格化 徐々に規模を拡大してシステム試験を実施

 性能安定性の鍵は、ファイルシステムの安定性に大きく依存 特に第２階層のLustreベースファイルシステムの安定性が重要

 Fugakuのファイルシステムの安定化：
Lustre開発コミュニティと密に連携
 引き続き、Bug fix、機能フィードバックに貢献していきます36

Copyright 2019 FUJITSU LIMITEDLAD19講演より抜粋

LNET Multi-rail Improvement
 Tatsushi Takamura and Shinji Sumimoto, Ph.D.
Next Generation Technical Computing Unit
FUJITSU LIMITED Sept. 24th, 201937

Copyright 2019 FUJITSU LIMITEDBackgrounds (FEFS IB Multi-rail) Fujitsu developed FEFS IB Multi-rail and operated on K computerand other HPC systems for over 7 years
FX10
FX100

K computer IB Multi-rail Features:

 High availability even if a single point of IB failure occurs
 High throughput by using multiple IB interfaces
 Various configurations•Not only Symmetric connections but also Asymmetric connections
node #A
ib0

ib1

IBSW

ib0

node #A
ib0

ib1

node #A
ib0

IBSW

ib1

node #B

ib0

node #B

ib1

ib2

node #A
ib0

ib1

IBSW

ib0

ib1

node #B

38

ib2

ib3

IBSW

ib0

ib1

node #B

Copyright 2019 FUJITSU LIMITEDBackgrounds (Lustre LNet Multi-rail) Lustre community is now developing similar Multi-rail features on
LNet level LNet Multi-rail, LNet Network Health, Etc…

 However the development is still going on…
 Therefore, we have evaluated the Lustre LNet Multi-rail assuming
the same features of FEFS IB Multi-rail In order to give feedback to current LNet Multi-rail implementation

39

Copyright 2019 FUJITSU LIMITEDFEFS IB Multi-rail (presented at LAD14)
 FEFS Approach: Add IB Multi-rail function
into Lustre network driver (o2iblnd).
 All IB I/F on the client can be used to communicate with a server.
 All IB connections are used by round-robin order.

 Continue communication when single point of IB failure occurs.
 All IB connections are used by round-robin order by each requests.

Multi-railby o2iblnd

Clients A

Clients B

Clients A

Clients B

Ib0

Ib0

Ib0

Ib0

Ib0

Ib0

IB SW
Failure
Ib0

Ib0

MDS/OSSMDS/OSS40

No failover
required

Copyright 2019 FUJITSU LIMITEDLustre LNet Multi-rail and LNet Health LNet Multi-rail: Introduced in Lustre 2.10(LU-7734)
 Using multiple interfaces including Ethernet and InfiniBand LNet Network Health: Introduced in Lustre 2.12(LU-9120)
 Detecting network failures of local interface, remote interface, network
timeouts and etc.
 Switching and resending among different interfaces
Flow of IB Multi-railFS

Flow of LNet Multi-rail
FSSendRequestptlrpcRetryLNet

ptlrpcLNet

LND Select(o2ib) Path

Send
Data

Notify SelectError Path

Send
Request

LND

Error

RetrySelect
Path

Send
Data

Notify
Error

Retry

Select
Path
Send
Data

Error

The basic idea is the same as FEFS IB Multi-rail(The difference is LND level or LNet level)41

Copyright 2019 FUJITSU LIMITEDOverview of LNet Health Detecting device status A local Network Interface (NI) is marked fatal,
if the device has gone into a fatal•ex. IB_EVENT_DEVICE_FATAL, IB_EVENT_PORT_ERR

 Maintaining health value

remoteHV1(remote)<HV2(remote) Each NI (both local, remote) has a health value (HV) HV is decremented when communications fail andincremented when succeeds

 Controlling path selection
 Selecting the healthiest local NI by HV•Fatal NI is removed from the candidates
 Selecting the healthiest remote NI which belong to
the same network which the local NI connected Communicating using the selected NIs42

failurelocal
HV1(local)

HV2(local)

Copyright 2019 FUJITSU LIMITEDSummary of Issues and Modifications
Issue

Description

Modification

No.1

Unable to detect IB hardware We handled IB hardware failure
failure (NI is not marked fatal). and a path is selected without
waiting for a HV decremented

No.2

No.3

No.4

Decrementing a health (HV)of normal NI
Unable to use Multi-rail on
asymmetric NIs

We set HV appropriately andreduced extra resending

After recovery of NI failure,
the NI is not used for a while(1000sec)

We stopped HV decrement atrecovery processing
to use the NI in a few seconds
after device recovery43We switched switch anothernormal NI to avoid for thesystem to become unusable

Copyright 2019 FUJITSU LIMITED44

Copyright 2019 FUJITSU LIMITED