Computer simulations create the future

Stable and Scalable Operationof Parallel File Systems at the K computer

Yuichi Tsujita
RIKEN AICS

Nov. 2, 2017@JLUG2017

RIKEN ADVANCED INSTITUTE FOR COMPUTATIONAL SCIENCE

Outline• Overview of the K computer and its file systems• Activities for stable and scalable operation• Alleviation of MDS load by using loop-back file systems• Elimination of client evicts• Alleviation of I/O interference by huge data accesses• Summary2Overview of the K computerand its file systems

Overview of the K computer
The K computer
Compute nodes# of CPUMemory Capacity

82,944
1.27PiB

6D Mesh/Torus Network (Tofu)

I/O Nodes

Users
InternetPre/Post
ServerFrontend
Servers

Local File System(LFS)(11PB)
Control & Management Network

Global I/O Network
Management
Servers

ControlServers

Stage‐in

InfiniBand(QDR)

Stage‐out

Global File System(GFS)(>30PB)
FEFS is used for both LFS and GFS.
(FEFS: Fujitsu Exabyte File System based on Lustre technology)4File systems at the K computer• Organization of file systems at the K computer– LFS : Performance oriented• for high performance I/O during computation– GFS : Capacity oriented• for huge data storing and high redundancy
File system

LFS

GFS *1

Total volume size~ 11 PB> 30 PB# volumes

111# OSSs

2,592

108

# OSTs

5,184

3,024

Disk system of OSTRAID5+0

RAID6
RAID6 FR (new three volumes only) *2*1 New three volumes have been operated since Apr. 2017.
*2 Extended RAID6 from Fujitsu (RAID6 FR) available for the new three volumes

5

LFS and I/O zoning•

Configuration of a LFS• I/O Zoning for LFS•

Compute nodes••I/O nodes
Local File SystemFile I/O is separated among jobs and processed by I/O nodes
located at Z=0. (OSS is running on I/O nodes.)Every OSTs are accessible from I/O nodes via Fibre Channel.
Z link is used for file I/O.

High I/O performance and low I/O interference

Z = 16

Z=2Z

Z=1
Z=0

X,Y,a,b,c

OSS

OSS

OSS

OSS

OSS

OSS
Fibre Channel

OSTs
6

Data-staging• Asynchronous data staging for high efficient job scheduling

OSS
LIO

GIO

FC
OST OST OST OST OST OSTIBOSS
LIO

OSS
LIO

OST OST OST OST OST OSTFC
LIO
LIO
LIO
OSS
OSS
OSS
Comp. Comp. Comp.

Compute node
GIO, LIOTime

BIO

BIO

((Active))((Standby)y)Stage‐InCabinet

Comp. Comp.

LFS

Tofu Interconnect

Comp. Comp. Comp.

System Rack
FrontendServers
IB SW
Stage‐OutGIO

BIO

BIO

((Active))((Standby)y)Comp. Comp.

System Rack
Job Execution( Job A))GFS

Job Execution( Job B))

Stage‐In( Job A))

Stage‐In( Job B))

Stage‐Out( Job A))

Stage‐Out( Job B))

GFS‐>LFS

GFS‐>LFS

LFS‐>GFS

LFS‐>GFS

7

Activities forstable and scalable operation•Alleviation of MDS load using loop‐back file systems•Elimination of client evicts•Optimization for alleviating interference by huge data accesses8High load of MDS (LFS)
Compute nodes which generated huge number of requests to an MDS of LFS

Physical y‐axis (Y=0~17)• Many file accesses(open, close, …)lead to high MDS load.
• High MDS load of LFS may affectmany applications accessing LFS.

• Providing loop‐back file systems(rank‐directory) to alleviate high
MDS load• Rank‐directory is recommended
for applications which access manyfiles and file per rank case.
ops in MDS
Physical x‐axis (X=0~23)(* Z=0 for I/O nodes)

192 compute nodes(2 system racks)

9

Rank-directory (loopback file system)< Storage space at LFS >/work

Meta-data

Shared directory Shared dir.

/JobName.JobID
Loopback /0
mount/1

ext2

ext2

Rank
directory/(n-1)/2

MDS

Object-data

Shared dir.

OSSsMeta-data

Rank dir.

ext2

ext2

Object-data

Rank dir.

process

Rank
#0

Rank
#1

Rank
#2

Rank
#(n-1)

Accesses to a shared directory
Accesses to a rank-directory• Reducing MDS accesses leads to effective utilization of LFS.
• I/O accesses in rank‐directories are free from slowdown of MDS performance.

10

Single node I/O performance evaluation by using IOzone
FEFS (shared directory among nodes) vs. loopback
Loopback outperformed FEFS for smaller data size with the help of filesystem cache.
read (64KB I/O block)write (64KB I/O block)

MB/s

MB/s

••

Bytes

Bytes

K. Yamamoto F. Shoji, A. Uno, S. Matsui, K. Sakai, F. Sueyasu, and S. Sumimoto,“Analysis and Elimination of Client Evictions on a Large Scale Lustre Based File System,” LUG’15

11Total metadata access performance••

Create 26K ops/node, unlink 37K ops/node by mdtest (100 files/node)
Rank directory (loopback) scales with a large number of processes.

about 21,000 times faster

K. Yamamoto, F. Shoji, A. Uno, S. Matsui, K. Sakai, F. Sueyasu, and S. Sumimoto,“Analysis and Elimination of Client Evictions on a Large Scale Lustre Based File System,” LUG’15

12

Impact for MDS load average•MDS CPU load
after1after2< MDS CPU load over time before and after loopback introduction through two steps(after1 and after2) >* Some large class job did not use loopback.

• MDS load average per hour: reduced to 1/3.5• Peak occurrence times per day (over 50%, 70%): reduced to 1/30

K. Yamamoto, F. Shoji, A. Uno, S. Matsui, K. Sakai, F. Sueyasu, and S. Sumimoto,“Analysis and Elimination of Client Evictions on a Large Scale Lustre Based File System,” LUG’15

13

Eviction problem• Eviction– File server evicts a client when a client does not work properly, e.g. noresponse to requests from servers.

• Impact of eviction– I/O accesses of running jobs on the node will fail.
• In many cases, jobs affected by evictions are aborted.

– Frequent evictions led to a serious decrease in node utilization.

14

Mitigation of evictions• Elimination of client evictions that we have done
– Step 1: Eliminating evictions during system board maintenance bysystem operation level– Step 2: Eliminating evictions during system board maintenance byimprovement of file system level• The two fixes reduced eviction occurrence ratio by a 1/72.
Eviction occurrence ratio/node
BeforeAfter

Improvements

0.47

0.0065

1/72

K. Yamamoto, F. Shoji, A. Uno, S. Matsui, K. Sakai, F. Sueyasu, and S. Sumimoto,“Analysis and Elimination of Client Evictions on a Large Scale Lustre Based File System,” LUG’15

15

Interference due to heavy data staging•Increase in response time in GFS accesses due to heavy data staging
Big increase in response times
100Lower is better
10.1
0.01

Stage-out for huge data set (stripe count=1)
0:00

23:00

22:00

21:00

20:00

19:00

18:00

17:00

16:00

15:00

14:00

13:00

12:00

11:00

10:00

9:00

8:00

7:00

6:00

5:00

4:00

3:00

2:00

1:00

0.001
0:00

write time (sec)10

Response times of 1MiB write from a frontend server to every OST in every 5 minutes on 2015/2/5

16

I/O interference in data staging•

Big performance degradation in file accesses from frontend servers due to hugescale of stage-out operations
LIO

LIO

LIO

FC
OST OST OST OST OST OST

Write operations using dd

IBStage‐InLFS
OST OST OST OST OST OST
FC
LIO
LIO
LIO

Frontendserver

GIO

Stage‐Out12IB SWContentionGIOGFSOSTs

1 stage‐out (default stripe count=1)
2 file accesses•

How do we mitigate performance degradation ?
1. I/O workload‐aware stripe count‐> Balanced I/O workload among OSTs
2. Load‐balancing among clients (QoS of FEFS)
17

I/O workload-aware stripe count•Tuning scheme of stripe count (Cs) in stage-out
N OST CS     N IO  k stg 

N OSS
N OST
N IOlthr
k stg
max
k stg

nstg max
 nstg , where    and k stg  min( N , k stg )IO N OSS  lthr 

The number of files that each OSS service thread manages
Maximum acceptable variance in I/O workload among OSTs
The number of OSSs
The number of OSTs
The number of I/O (GIO) nodes
Maximum number of service threads on each OSS
The number of files in staging at each GIO
Maximum number of files that one GIO can manage

Y. Tsujita, T. Yoshizaki, K. Yamamoto, F. Sueyasu, R. Miyazaki, and A. Uno, “Alleviating I/O Interference ThroughWorkload‐Aware Striping and Load‐Balancing on Parallel File Systems,” Proceedings of ISC’1718

QoS in GFS accesses•QoS function of FEFS utilized for data staging at the K computer

GIO nodes
Staging

I/O

nstageGFSthreads

nstage /(nstage  n front )n front /(nstage  n front )Frontend servers

n frontthreads

FEFS(OSSs, OSTs)

Limiting the maximum number of service threads for each client‐> Guarantee I/O bandwidth for each client

19Performance improvements in GFS accesses with QoS function
96 GIOs(12x24x2), 576 files (12GB/file)
SIN

SOT

Cs=2w/o QoS

Cs=1

Cs=12

Cs=32

Our model predictedthat Cs=14 was thebest.

Performance
evaluation showedthat Cs=12 was thebest.

Improvements

w/ QoS (20:80)

Cs=1

Cs=2

QoS function wasturned out to beeffective in I/Ointerference
mitigation.

Cs=12
Cs=3220

Contribution to Lustre development by Fujitsu•

Incorporated function derived from R&D works for the K computer by Fujitsu
Jira 1

Function

Landing

LU‐2467

Ability to disable pinging

Lustre 2.4

LU‐2466

LNET networks hashing

Lustre 2.4

LU‐2934

LNET router priorities

Lustre 2.5

LU‐2950

LNET read routing list from file

Lustre 2.5

LU‐2924

Reduce ldlm_poold execution time

Lustre 2.5

LU‐3221

Endianness fixes (SPARC support)

Lustre 2.5

LU‐2743

Errno translation tables (SPARC support) Lustre 2.5

LU‐4665

lfs setstripe to specify OSTs

Lustre 2.7

1. https://jira.hpdd.intel.com/projects/LU/issues/

21Summary• Our efforts about FEFS as shown below have led to high availability andhigh I/O performance.
1. Loop-back file system
2. Eviction treatment,
3. Stripe count tuning and QoS function, and so forth• Further efforts for high availability of file systems are in progress.

22

Acknowledgment
Special thanks to• RIKEN AICS
• F. Inoue, M. Iwamoto, F. Shoji, K. Sugeta, A. Uno, K. Yamamoto• FUJITSU Limited• Y. Furutani, H. Hida, N. Ikeda, S. Matsui, R. Miyazaki, M. Okamoto, R.
Sekizawa, F. Sueyasu, S. Sumimotofor giving many useful information about their efforts described in thispresentation.

23