JLUG2015

Topics from LUG-2015 and LAD &Lustre Development Summit 2015
Oct. 22 2015
Shinji Sumimoto (Fujitsu Limited)

Copyright 2015 FUJITSU LIMITEDOutline of This Talk
 From LUG-2015
 Metadata Access Reduction of Large Scale Lustre Based
File System

 From Lustre Development Summit 2015
 Fujitsu Session: Toward Exascale Computing

1

Copyright 2015 FUJITSU LIMITEDLUG2015

Metadata Access Reductionof Large Scale Lustre Based
File System
Apr.14 2015
Shinji Sumimoto*, Shuji Matsui, Kenichiro Sakai, andFumichika Sueyasu (Fujitsu Limited)
Fumiyoshi Shoji, Atsuya Uno, Keiji Yamamoto (RIKEN AICS)

Copyright 2015 FUJITSU LIMITEDOutline File System Usage in User Jobs on K computer File Access Issues on Local File System
 Meta Data Access Distribution by Loopback File System
 Evaluation on K computer

3

Copyright 2015 FUJITSU LIMITEDFILE SYSTEM USAGE IN
USER JOBS ON K COMPUTER
4

Copyright 2015 FUJITSU LIMITEDOverview of FEFS for K computer Goals: To realize World Top Class Capacity and Performance
File system 100PB, 1TB/s
 Based on Lustre File System with several extensions• These extensions are now going to be contributed to Lustrecommunity.
 Introducing Layered File system for each file layer
characteristics
 Temporary Fast Scratch FS(Local) and Permanent Shared FS(Global)
 Staging Function which transfers between Local FS and Global FS
is controlled by Batch Scheduler
FileServerFileServerLocal File System(work temporary)

For Performance

Staging

Local File System
FileServerFile
Global File System

Cluster File System
FEFS

Configuration of FEFS
for K computer
5For Easy Use
For Capacity andReliability
Global File System(data permanent)
2011 FUJITSU LIMITED
Copyright 2015

Job Execution and File System Accesseson K computer
82,944 Compute Nodes
4.Stage-in
Processing
5. Job
Processing

Portal Server

6.Stage-out
Processing

65I/O

3Login Nodes

Automatic Stage-inand Stage-out by
Batch Scheduler

Local
File System
Local FS 11PB,
1 MDS+ 2,592
OSSes (5,184 OST)
3.2TB/s Read,
1.4TB/s Write

ＧＷ

4

Global FS 30PB,
5 MDS+ 90 OSSes(2,880 OST)
0.2TB/s Read/ Write

Global File System

Job Dispatch

1/data/home

Job Dispatching
on a Login Node

Stage-In
Files
Program
Stage-Out
Files

Batch Scheduler

Users use/home and /data66

2Job Execution
EnvironmentCopyright 2015 FUJITSU LIMITEDFILE ACCESS ISSUES ON
LOCAL FILE SYSTEM
7Copyright 2015 FUJITSU LIMITEDMeta Data Access Issues of Local File
System on several 10,000 node job.
 Creating a lot of files per MPI rank at a time.
 1,000 file per rank creation becomes 10 M file creation per job.
 Creating and deleting files take several hours to finish and cause
slow MDS response
Compute nodes

ClientI/O NetworkMDSOSSFile Servers

 Execution binaries on shared directory.
 Concentration access to a single MDS and OST from several
10,000 node takes a long time to finish. Long time delay occurs
on starting jobs
Compute nodes
ClientI/O NetworkMDSOSSOSS

File Servers
8

Copyright 2015 FUJITSU LIMITEDMDS CPU Load on Dec. 2013.
 MDS Load was average 24% peak 100% on Dec. 2013.

2013/12/08-12/14
L-MDS
Load (CPU)Max
100% 1009080

Load (%)

7060
50
40

Ave 3024% 209

2013/12/14

2013/12/13

2013/12/12

2013/12/11

2013/12/10

2013/12/90

2013/12/8

10

Copyright 2015 FUJITSU LIMITED

Issues to Solve and Our Goals
Issues to Solve:
Concentration access to a single MDS or OST on jobexecution
Violent Fluctuations of MDS/OSS load depending onjobs

Our Goals
Distributing and leveraging Meta Data and Data AccessProviding faster access performance per MPI rank

10Copyright 2015 FUJITSU LIMITEDMETA DATA ACCESSDISTRIBUTION BY
LOOPBACK FILE SYSTEM
11

Copyright 2015 FUJITSU LIMITEDMeta Data Access Distribution by Loopback
File System on K computer Providing real local file system per rank by using loopback file Creating loopback file and mounting it as Ext2 file system per MPI rank

 Rank local data and execution binaries are copied to rank local file system

 Job scheduler software automatically manages creating, mounting
and deleting the rank local file system.
 MDS load can be decreased to only one file creation/deletion per rank

 No fluctuation and no dependence per Job types (Constant Load)
Storage Location
4 Job

Z Axismount05

61

2

7
3

Ext2 FS on loopback files
OSS

Ext2 FS

file
fileOSS

file
file

file
file

file
file
12

Copyright 2015 FUJITSU LIMITEDComparison of Multiple MDS vs. Loopback We compare the loopback with multiple MDS which
could be the other method to solve high load ofMDS.
 Multiple MDS(Lustre DNE)
 Pros:•Increasing Meta Data performance on shared file system Cons:•Requiring additional hardware resource: MDS, MDT
Scalability is limited to hardware resource Loopback Pros:•Completely Scalable Meta Data performance for rank local access•No additional hardware

 Cons:•Unable to share among the other nodes•Additional Ext2 file system and Loopback Layer Overhead
13

Copyright 2015 FUJITSU LIMITEDEVALUATION ON
K COMPUTER
14

Copyright 2015 FUJITSU LIMITEDEvaluation of Loopback Based Rank Local
File System on K computer Single Node File Access Performance Total Meta Data Access Performance Comparison of MDS Load (Before vs. After)

15

Copyright 2015 FUJITSU LIMITEDSingle Node File Access Performance Single Node File Write/Read Performance by iozone
 Loopback based file system achieved better performanceat small file size by file system cache

MBytes/s

Iozone Read Performance (64KB IO Block)

MBytes/s

Iozone Write Performance (64KB IO Block)

Bytes

Bytes

1GB16

1GB

Copyright 2015 FUJITSU LIMITEDTotal Meta Data Access Performance Loopback Based Local FS Dramatically Scales over 10,000 Nodes!
 Create 26K ops/node, unlink 37K ops/node by mdtest 100 files/node Providing higher constant meta data access performance for each node453M ops
21,571 times Faster
unlink@9216 node

21K ops

17

Copyright 2015 FUJITSU LIMITEDMDS CPU Load Comparison (Before vs. After)

Longtime evaluation except maintenance time(2013/12-2015/2)
 MDS Load average per hour: about 1/3.5
 Peak occurrence times per day(Over 50%,70%): less than 1/30
Some Large ClassJobs did not apply
Rank Local
File System

After1After2

Need to add someoption to use
loopback basedrank directory

Before-13/12/27

Average MDS
Load %

After1:-14/6/29

After2:-15/2/28

After(All)

25.1

8.21

6.36

7.13

Over 50%

2.32

0.12

0.02

0.06

Over 70%

0.68

0.04

0.00

0.02

times per day
Times per day

18

Copyright 2015 FUJITSU LIMITEDSummary Meta Data Access Distribution by Loopback File SystemDistributing and leveraging Meta Data and Data AccessProviding higher constant access performance onrank local file

Evaluation Achieved Better File Access Performance up to 128MB
 Loopback Based Local FS Dramatically Scales over 10,000 Nodes!
 MDS Load average: about 1/3.5
 Peak Occurrence Times per Day: less than 1/30

 Introduction of Loopback based rank local file system isvery effective on K computer operationeven if 1 MDS+ 2,592 OSSes (5,184 OSTs) file system.
19

Copyright 2015 FUJITSU LIMITEDLuster Developer Summit 2015

Toward Exascale Computing
Yuta Higuchi,
Shinji Sumimoto (Fujitsu Limited)

Copyright 2015 FUJITSU LIMITED

Storage and System Requirement from theArchitecture Roadmap (IESP 2012@Kobe)

21

Copyright 2015 FUJITSU LIMITEDExascale File System Design
Trade off: Power, Capacity, Footprint, Costs
Difficult to reach1EB and 10TB/s class file system on single filesystem in limited power consumption.

Third Storage layer for Capacity is needed:Three Layered File System
Local File System for PerformanceGlobal File System for Easy to Use
Archive File System for Capacity

Local File System
Global File System

Archive File System

22

Copyright 2015 FUJITSU LIMITEDThe Next Integrated Layered File System Architecturefor Exascale Systems (Presented at LUG 2013/Panel) Local File System(10PB Class): ex: Memory , SSD Based, etc..
 Application Specific, Existing FS, Object Based, etc..
 Global File System(100PB Class): ex: Disk Based, etc..
 Lustre Based, etc..
 Archive File System(1EB Class): ex: HSM(Disk+Tape) etc..
 HSM, Lustre, Cloud, other file system
ComputeNodes
Nodes
ComputeCompute
Nodes
Compute Nodes

Thousands of Users
LoginServerHigh Speed for Application

Lustre Based
Lustre Based
Ext[34]
Based
LustreBased
Ext[34]Specific
Based
Application
ObjectBased
Ext[34]
Based
Object Based
Application
Specific
Object Based
Existing
ApplicationFSSpecific
Application SpecificObject BasedSharedUsability

Lustre Based

OtherOrganization

OtherSystems

High Capacity & Redundancy& Interoperability

HSM, Cloud, Other FS

Transparent Data Access

Job Scheduler
23/dataCopyright 2015 FUJITSU LIMITED

Issues of File System for Exascale Systems
Discussed at last Summit 2014
 System Limits: Increase the logical upper limits (capacity, # of clients, # of OSTs, etc…)
 Memory Usage: Required memory should not proportional to # of OSTSs
 Meta Data Performance: Reduce metadata access. Lustre DNE improves metadata
performance, but requires additional hardware resource, MDS and MDT. So, scalability islimited to hardware resource I/O Throughput and Capacity: Achieve higher throughput (10TB/s~) and larger capacity(~1EB) in limited power consumption and footprint System Noise: Eliminate OS jitter to maximize performance of massively parallel
applications•Will Be Discussed Today Power Consumption: Reduce power consumption of extreme large storage systems
 Dependability: Data must not be lost even if storage(RAID) failure, and operations
should be resumed quickly
 Eviction:

24

Copyright 2015 FUJITSU LIMITEDExascale Concerns (Summit 2015)
 Power Consumption
 Concern: Reduce power consumption of extreme large storage systems
 Approach: Introduce low power device in hierarchical storage system• e.g. SSD for 1st layer (fast job I/O area), Tape device for the bottom layer (archive area)•And stopping hardware such HDDs in the storage devices, part of OSSs, etc• MAID for HDD (MMP prevents to use this) Dependability
 Concern: Data must not be lost even if RAID storage gets defective, andoperations should be resumed quickly• e.g. controller module failures, defective lot of disks, software bug, etc…
• e.g. Running “lfs find” to find affected files takes a long time ..
• e.g. Running fsck on the storage cloud take a month.

Å Good idea, but RAID1 requires
 Approach?: OST-level RAID(LU-3254 by Jinshan)doubled space. RAID-5 maybe?

 One Approach: File Services should not be stopped even if some storages areoffline.
25

Copyright 2015 FUJITSU LIMITEDExascale Concerns (Summit 2015) (Cont’d)Eviction
 Background:• Since OBD ping causes OS jitter, we have suppressed it.
(LU-2467)• Introduced alternative solutions (e.g. cooperation with hardware), but still haveproblems with notifying the clients their evicted status.
(reported at LUG2015)

 Concern: The clients do not notice if they are evicted until they do file I/O,
which ends up EIO.

26

Copyright 2015 FUJITSU LIMITEDExascale Concerns (Summit 2015) (Cont’d) Approach:
1.
2.
3.

When a server evicts a client, the server notifies MGS
MGS notifies the evicted client to connect the server
The client sends ping request to the server(LU-6657: eviction notifier)(1)Evict client
MDS or OSSping & reconnect(3)Send ping request
for reconnecting

TargetMGC

ClientHandler

notify  clientʼ’s
NID obd_name

MGS
MGT

MGC

(2)Send reconnect
message to client

obd_name
27

Copyright 2015 FUJITSU LIMITED議論論：Dependabilityについて
 Lustre・FEFSはディスク故障は起きない前提で設計されている
 RAIDやミラーリング、サーバ故障時もFailover時にディスクアクセスが継続

 しかし、稀にストレージ故障によりデータが失われる場合が発⽣生しうる 2台同時故障 on RAID5, Disk Firmware問題など、、

 Lustre・FEFSのデータ復復旧は2段階で実⾏行行
 Backend Filesystemのfsck
 OSTのfsck：lfck

 OSTの規模が⼤大きくなると復復旧時間が課題
 Backend Filesystem復復旧中はファイルシステムアクセス不不能
 どのファイルのデータが失われたかを知るには全ファイルのスキャンに時間がかかる

 復復旧時間の短縮に加え、短時間でのサービス再開が求められている
 Backend Filesystemの復復旧ができればサービス復復旧は可能
 しかし、RAID復復旧不不能の時点でBackend Filesystemは⼤大きなダメージ ⼀一部のOSTがOffline状態でもサービス復復旧できるほうがよいケースがあり得る
28

Copyright 2015 FUJITSU LIMITED議論論：Dependabilityについて(2) RAID故障の対策：RAID故障は本来は起きてはならない システム設計上はストレージで対策すべき問題
 原則、更更なる多重化しかないのが現状
 対策の現状： 機器コスト、性能、運⽤用停⽌止時間を含め総合的に考える必要

 OSTデータの多重化
 全データ多重化•OST毎多重化： RAID1, RAID1+6, 5+6, 6+6•HSM機構を活⽤用した多重化：バックアップとしてHSM機構を活⽤用

 部分データ多重化•ファイルレベル多重化： ファイルを選択して多重化•Backend Filesystemメタデータのみ多重化：復復旧⾼高速化、容量量は節約可 多重化レベル•ストレージを含むBackend Filesystemレベル•多重化に対応したOST： Spare OSTの提案

 今後も引き続きLustreコミュニティと⽅方策を議論論していく
29

Copyright 2015 FUJITSU LIMITEDCONTRIBUTION STATUS
AND PLAN
Copyright 2015 FUJITSU LIMITEDFujitsu Contributions until 2014
 Fujitsu have submitted Lustre enhancements with Intel.
JiraFunction

LandedLU-2467

Ability to disable pinging

Lustre 2.4

LU-2466

LNET networks hashing

Lustre 2.4

LU-2934

LNET router priorities

Lustre 2.5

LU-2950

LNET read routing list from file

Lustre 2.5

LU-2924

Reduce ldlm_poold execution time

Lustre 2.5

LU-3221

Endianness fixes (SPARC support)

Lustre 2.5

LU-2743

Errno translation tables (SPARC Support)

Lustre 2.5

LU-4665

lfs setstripe to specify OSTs

Lustre 2.7
31

Copyright 2015 FUJITSU LIMITEDFujitsu Contributions in 2015 (1) We are submitting new features for Lustre.
JiraFeature

Submission Status

LU-6531

Fujitsu's o2iblnd Channel Bonding Solution(IB multi-rail)

In Review

LU-6657

Eviction Notifier(Automated Eviction Recovery)

In Review

LU-6658

single stream write performance
In Review
improvement with worker threads in llite(Single Process IO Performance Improvement)

32

Copyright 2015 FUJITSU LIMITEDFujitsu Contributions in 2015 (2) We are submitting bug-fixes for Lustre as well.
JiraPatch

Submission Status

LU-6600

Race lustre_profile_list

Landed to Lustre 2.8

LU-6624

LBUG in osc_lru_reclaim

Landed to Lustre 2.8

LU-6643

write hang up with small max_cached_mb

In Review

LU-6732

Cannot pick up EDQUOT from ll_write_begin
and ll_write_end

In Review

33

Copyright 2015 FUJITSU LIMITEDFujitsu Contributions in Future Fujitsu will continue submitting new features.
FeatureSubmission Schedule

Directory Quota

2nd half of 2016

Client QoS

2nd half of 2016

Server QoS

TBD

Memory Usage Management

TBD

Snapshot

TBD

 We will also submit Lustre 2.x bug-fixes in 2016.

34

Copyright 2015 FUJITSU LIMITED35

Copyright 2015 FUJITSU LIMITED