Computer simulations create the future

Improving Meta-Data Access Performance
Towards High Availabilityof Large Scale Parallel File Systems
Yuichi Tsujita
RIKEN Center for Computational Science

JLUG2018 (Oct. 25, 2018@Tokyo)

RIKEN Center for1Computational ScienceOutline• Background• K computer and Its Filesystems• QoS Management for Load-Balancing• Performance Evaluation• Summary2

Background (1)• Mission of Operation and Computer Technologies Division, RIKEN
Center for Computational Science– High level service for users at the K computer• High availability• Stable operation– Stable and scalable operation of parallel file systems• Many challenges have been done towards high level service.
– Alleviation of MDS load by using loop-back file systems– Elimination of client evicts– Alleviation of I/O interference during heavy data accesses, etc.
There are still performance issues towards high quality service for users!

3

Background(2)• High MDS load or very slow MDS response in a huge scale of parallel
file systems at the K computer lead to performance degradation in– local I/O by compute nodes
Ineffective operation– data staging, and so forth• How can we alleviate high MDS load / slow MDS response forfurther performance improvements?
• Knowing the root-causes of such high MDS load / slow MDSresponse• Adopting QoS control for an MDS, which is available on the parallel
file systems at the K computer Effective operation
4

System Configuration of the K computer
40 m x 40 m
Full System

4000mm x 800mm Compute Rack × 864
800mm x 800mm
Compute Rack

2 Cabinets
Compute Rack × 4
Disk Racks × 1

SB×24
IOSB×610.6(11.3)PFLOPS
1.27(1.34)PiB500mm x 500mm
System Board(SB)Node×449.2(52.4)TFLOPS
6.00(6.38)TiB

NodeCPU×1ICC×1memory
12.3(13.1)TFLOPS
1.50(1.59)TiB
512GFLOPS
64GiB
128GFLOPS
16GiB( ) included IO node performance and memory capacity.
5File Systems of the K computer
The K computer
Compute Nodes# of CPU
82,944
Memory Capacity 1.27PiB

Users
Internet

6D Mesh/Torus Network

Pre/Post
ServerI/O Nodes

Frontend
Servers

Local File System(LFS)(11PB)
Control & Management Network

Global I/O Network
L-MDS Management Control
Servers

MDS for LFS

Servers

Stage-In

Stage-Out

Global File System(GFS)(>30PB)

FEFS is used for both LFS and GFS.
(FEFS: Fujitsu Exabyte File System based on Lustre technology)
6File System Configuration• Organization of file systems at the K computer– LFS : Performance oriented• for high performance I/O during computation– GFS : Capacity oriented• for huge data storing and high redundancy
File system

LFS

GFS *1

Total volume size

~ 11 PB> 30 PB# volumes

111# OSSs2,592

108# OSTs

5,184

3,024

Disk system of OST

RAID5

RAID6
RAID6 FR (new three volumes only) *2*1 New three volumes have been operated since Apr. 2017.
*2 Extended RAID6 from Fujitsu (RAID6 FR) available for the new three volumes

7

Rank-Directory (Loopback File System) at the LFS• Mitigation of I/O interference among processes< Storage space at LFS >/work

Meta-data

Shared directory Shared dir.

/JobName.JobID
Loopback /0mount

/1

ext2

ext2

MDS

Rank
directory/(n-1)/2

Object-data

Shared dir.

OSS

Meta-data

Rank dir.

ext2

ext2

Object-data

Rank dir.

Process Rank

#0

Rank
#1

Rank
#(n-1)

Rank#2

Accesses to a shared directory
Accesses to a rank-directory• Reducing MDS accesses leads to effective utilization of LFS.
• I/O accesses in rank-directories are free from slowdown of MDS performance.

8Data-Staging at the K computer• Asynchronous data-staging
Compute Node
GIO, LIOTime

Job Execution( Job A)

Job Execution( Job B)

Stage-In( Job A)

Stage-In( Job B)

Stage-Out( Job A)

Stage-Out( Job B)

GFS->LFS

GFS->LFS

LFS->GFS

LFS->GFS

Overlaps between
job executions
and data-staging
9 Stage-in phase includes rank-directory creation.
9 High MDS load or quite slow MDS response may increase times forrank-directory creation.
An increase in times for stage-in phase
9 Ineffective job scheduling9High MDS Load/Slow MDS Response• MDS Activities

High MDS load (~ 23 hours)

Quite low
MDS load

10

MDS Request Queue Status•

High MDS load•

Slow MDS response••

Full in request queue
Very slow operation

11

Interference due to MDS Problems• High MDS load/Slow MDS response lead to9I/O performance degradation in local I/O at the LFS
9Increase in times for data-staging, and so on.
• High MDS load
• Fully utilized service threads for dominant MDS heavy job accessinga large number of files concurrently• Quite slow MDS response• Every service thread was blocked to wait responses from associatedOSSes.

Alleviation in I/O interference

QoS control for service threads at the MDS
12

QoS of FEFS(1)• Requests in queue under QoS control
QoS: Group-B

QoS: Group-A
index, index,etc. etc.
Queue of job-1index, index,etc. etc.
Queue of job-2

In-direct reference
req

ptlrpcd
request queue req

req

req

req

ll_mdt_xxxx

13

QoS of FEFS (2)• Load-balancing by QoSGroup-AI/O

𝑅𝑅𝐴𝐴
𝑛𝑛𝐴𝐴 = 𝑛𝑛𝑆𝑆 × () threads
100RA (%)

𝑅𝑅𝐵𝐵
𝑛𝑛𝐵𝐵 = 𝑛𝑛𝑆𝑆 × () threads
100FEFS(MDS, OSS)

I/OGroup-B

𝑅𝑅

RB (%)𝑅𝑅𝐴𝐴𝐵𝐵
𝑛𝑛𝐴𝐴 = 𝑛𝑛𝑆𝑆 × (100), 𝑛𝑛𝐵𝐵 = 𝑛𝑛𝑆𝑆 × (100)(GIO nodes : Compute nodes) = (100 × n A
( n A + nB )14:

100 × nB)
( n A + nB )

Performance Evaluation• Stripe count impact in MDS performance• QoS impact in fair-share execution among concurrent running jobs
• QoS impact in data-staging

15

MDTEST performance of L-MDS1,536 processes@4x6x8(8 processes/node)• 2 sets (768 and 1,536 processes on 192 compute nodes) of MDTEST runs
• Command: ./mdtest -d ../md_dir –n 100 –i 3 –F –u• Seven sets of stripe counts (Cs): 1, 2, 4, 8, 12, 24, and 48 (12 is default configuration.)• Mean values and standard deviations from 6 iterations

768 processes@4x6x8 (4 processes/node)

Cs: stripe count

9 An increase in stripe count led to performance degradation, especiallyin “File creation,” “File stat,” and “File removal.”
16

QoS for I/O Interference Alleviation• QoS control at the MDS– Managing service thread assignment for several groups,
such as local file I/O and data-staging– Fair-share job execution in order to mitigate I/Ointerference each other17

Examination of Fair-Share Execution(1)• I/O interference impact among two concurrent jobs (MDTEST runs)

JOB-B(np=768,Cs=4)

71% for file I/O (17 threads)

MDS– JOB-A(6,144 processes): ./mdtest -d ../work –n 100 –i 200 –F –u– JOB-B(768 processes): ./mdtest -d ../work –n 100 –i 3 –F –u

29%for others(5 threads)
JOB-A(np=6,144,Cs=96)

Executed jobs

768

np96

4Cs71%None

QoS rate

None

Fair-share

GIONotation
JOB-B

6,144• GIO group and compute node grouputilized up to 29% and 71% of 24 service
threads of the MDS.
• Fair-share function split available service
threads evenly among JOB-A and JOB-B.
• Performance measurements at JOB-B

reference*

JOB-A

4

Fair-share: on(50%:50%)/off

off

768

96

JOB-A:JOB-B=50%:50%(up to 90% each if available)

None
JOB-B

6,144

471%

JOB-A768on

JOB-B* The number of service threads was arranged to be 17 for MDTEST runs.
18

Examination of Fair-Share Execution (2)• MDTEST results– Big performance degradation without fair-share control (“off”)– Big interference mitigation under fair-share control (“on”)

35000
30000
25000
20000
15000
10000
5000
0
File creation

File statreference

19offonFile read

File removal• Although performance degradation was observed compared with the“reference” case, improvement ratio relative to the “off” case wasbigger than minimization ratio relative to the “reference” case.

OPSExamination of Fair-Share Execution(3)• CPU utilization at the MDS– Higher CPU utilization around 70% was realized under fair-share control.

• Similar CPU utilization compared to the “reference” case (~70%)

Low CPU utilization due to contention incommunication with associated OSSes.

– Without fair-share control, CPU utilization was low (around 20%).

20

QoS Impact in Data-Staging (1)Figuring out time for rank-directory
creation: 𝑡𝑡
𝑟𝑟𝑟𝑟𝑟𝑟_𝑑𝑑 = 𝑡𝑡𝐸𝐸 − 𝑡𝑡𝑆𝑆

Picking up the most earliest time stamp tostart ( 𝑡𝑡𝑆𝑆 ) and the most slowest time stampto end ( 𝑡𝑡𝐸𝐸 ) for rank-directory creation• Evaluation of I/O Interference impact in data-staging
1. Submit a job for data-staging
2. Start-up of stage-in phase
3. Rank-directory creation at each rank on OSTs
4. Start-up of data transfer from GFS to LFS
5. End of stage-in phase21

QoS Impact in Data-Staging (2)• Times for rank-directory creation in data-staging (stage-in phase)– Cs=96– Measurement of stage-in times under MDS high load due to an MDTEST run by
6,144 processes on the same number of compute nodes.

• QoS off: Rank-directory creation was not completed within 5 minutes.
• QoS on: Comparable in times for rank-directory creation of “Reference” casewithout the MDTEST run¾ QoS management has much impact in interference alleviation.
22

Summary• We have investigated root-causes of (1) high MDS load and (2) quite slow
MDS response at the K computer.
– High MDS load was due to a large number of concurrent file accesses.
– Quite slow MDS response was caused by a large number of concurrent
file accesses under a large stripe count.

• Large stripe count configuration caused congestion on associated OSSes.
• Service threads at an MDS were unable to proceed new requests due to slow
response from associated OSSes.

• QoS control at the MDS has been introduced to mitigate such MDS
performance degradation.
– I/O interference among user jobs was mitigated under fair-share service
thread allocation for each job.
– Minimization in times for rank-directory creation was achieved even if
an another job which caused high MDS load was running at the sametime.
– QoS management for the MDS has been introduced about six months,
and high availability has been achieved in our operation.
23

Acknowledgment

Special thanks to• RIKEN Center for Computational Science• F. Inoue, M. Iwamoto, F. Shoji, K. Sugeta, A. Uno, K. Yamamoto• FUJITSU LIMITED• Y. Furutani, H. Hida, N. Ikeda, S. Matsui, K. Miyamoto,R. Miyazaki, M. Okamoto, R. Sekizawa, F. Sueyasu

for giving many useful information about their efforts
described in this presentation.

24