JLUG2014, Oct. 17, 2014, Tokyo, Japan

MPI-IO Performance Improvement on FEFS

Yuichi Tsujita (RIKEN AICS)

Outline
1.

2.
3.
4.
5.
6.
7.

Introduction
MPI-IOFEFS and Tofu interconnect in the K computer
Optimizations for FEFS
Performance impacts of optimizations
Remarks from my personal experience
SummaryIntroduction (1)Our focus on parallel I/OUpper layer
Parallel Computing Application
Application oriented I/O interface (HDF5, PnetCDF, …)Middleware (MPI (MPI-IO), OpenMP, …)Parallel File System (Lustre, FEFS, PVFS2, …)Lower layer

Introduction (2) HDF5, PnetCDF
 Widely used application oriented I/O libraries (HDF5,PnetCDF, …)
 MPI-IO is used in underlying parallel I/O layer
 MPI-IO performance improvement is essential to cope with ahuge scale of data management.
 Performance Improvement of MPI-IO on Lustre/FEFS Optimization of an MPI-IO implementation named ROMIO One of our target platforms => K computer (FEFS with Tofu)
 Optimizations suitable for FEFS striping with topology-awareness forTofu interconnect

MPI-IO (1) Collective I/O
MPI_COMM_WORLD
Data transfer

MPI_COMM_WORLD

MPI_File_write_all

High aggregated throughput
writeBottlenecka. Independent I/O• Large overhead in datacommunications• Bottleneck in the
representative process
to perform local I/O

Parallel file system (e.g., Lustre, PVFS2, …)
b. Collective I/O• High throughput by using parallel I/O• Suitable for parallel file systems such asLustre

PDP 2014, February 13, 2014

MPI-IO (2) Typical MPI-IO access patterns
 APIs are based on the MPI standard.

We are focusing on this pattern.

6

MPI-IO (3) ROMIO
 Widely used MPI-IO implementation available in MPICH orOpenMPI
 supports many kinds of file systems such as Lustre, FEFS,
PVFS2, and so forth

Parallel I/O
in upper layer

HDF5, PnetCDF, …
MPI-IO (ROMIO)ROMIOADIO common layer
ad_fefsad_lustre

ad_pvfs2…

File system (Lustre, FEFS, PVFS2, …)

ADIO

MPI-IO (4)● Two-phase I/O (TP-IO) in collective write for non-contiguous file accesses

Every file access is aligned to a collective buffer (hereinafter, CB).
 Repetitions of file accesses and data exchanges
data region to be written

FileFile read(contiguous)File domain of process
0
Read
Collective buffer on process 0User buffer on process 0

Dataexchange

Collective buffer on process 1
User buffer on process 1

Data exchange and modification of collective bufferCollective buffer on process 0
File write(contiguous) File

File domain of process
1
Read

Write

Collective buffer on process 1
Write

FEFS and Tofu in the K computer (1) Computing nodes with the Tofu interconnect

1 Tofu-unit(2x3x2 nodes)

1 system board(4 computing nodes)

6D mesh/torus using the Tofu interconnect
・Axis : X, Y, Z, a, b, c・X,Z,b : torus (Z=0: I/O node), Y, a, c : Mesh・Network size : (X, Y, Z, a, b, c) = (24, 18, 17, 2, 3, 2)

FEFS and Tofu in the K computer (2)Network configurations of a local file system
Computing Nodes

I/O Nodes
Local File System• I/O nodes are located on z=0 of z-axis.
• Every OSTs are accessible from I/O nodes via Fibre
Channel interconnects• Accessing a target OST through an OSS on an I/O
nodez = 16…

z=2Z-axis

z=1
OSS

OSSOSS

OSS

OSS

OSS

z=0

X,Y,a,b,c-axis
OSTs

Fibre Channel

Optimizations for FEFS (1) FJMPI (based on OpenMPI) with Fujitsu’s enhanced implementations suitable forthe K computer
 MPI-IO is available by using ROMIO enhanced forFEFS.
 ADIO for FEFS supports file accesses to FEFS.

Optimizations for FEFS (2)Current implementation may have the following problems.
Network contention among aggregator processes and OSTs Performance degradation due to many I/O requests on each OSTFile viewProc. 0
Proc. 1
Proc. 2

Partitioned access pattern
File domain of process 0

File domain of process 1

File domain of process 2

Stripe size

Aggregator process 0

Aggregator process 1

Aggregator process 2

Contention
OST #0OST #1

OST #2

So many I/O requests lead to performance degradation.

Optimizations for FEFS (3)Improper aggregator mapping for FEFS and Tofu
Current: Simply mapping from rank=0 in a sequential orderz-axisof Tofu

rank-ID

MPI process@Computing
nodes
OSSs@
I/O nodes
Storage(OSTs)

6789101118

19202122

23

0123451213141516

17 z=1

Contention
So many I/O requests lead to performance degradation.
Process mapping in X->Y->Z order for 2x3x2 comp. nodes• 12 OSTs• All the MPI processes are I/O aggregators.

z=2z=0Optimizations for FEFS (4)Simplified communication for lesser contention
 Reduction of the number of I/O requests on each OSTFurther optimization

Grouping aggregators which access the same OSTon the same z-axis

MPI process@Computingnodesz-axis ofTofu

6

7

8

9101118

1920

21

22

23

0

1234

51213

14

15

16

17 z=1OSSs@
I/O nodesStorage(OSTs)

z=2z=0

Process mapping in X->Y->Z order for 2x3x2 comp. nodes• 12 OSTs• All the MPI processes are I/O aggregators.

Optimizations for FEFS (5)Striping pattern based aggregator process mapping
z-axis of Tofu

Computing
Process

I/O aggregator mapping
6
0

7
1

8
2

9
3

4

5

OSS@I/O nodes
OST• I/O aggregator mapping• based on striping access patterns against OSTs
• independent of the rank number order．．．

151

Optimizations for FEFS (6)Service threads on OSS
Maximum size of dirty page: 4 MiB
 Maximum number of rpc requests: 8
 Maximum page size of each rpc request: 512
 The maximum number of service threads: (ost, ost_io) 32＜ O_DIRECT ＞＜ Buffered I/O ＞• Many I/O requests to each OST => Performance degradation• I/O Throttling alleviates I/O request contention and leads to performance
improvements

Optimizations for FEFS (7)I/O throttling in write phases of two-phase I/O

MPI rank

Readphase

Exchange phase

ReadphaseWritephase

Exchange phase

time
MPI_Waitall()

MPI_Irecv()/
MPI_Isend() (every process)•Every local group manages I/O throttling to keep maximum number of I/O requests toevery OST at the same time.

Optimizations for FEFS (8) Current implementation status FEFS striping pattern oriented aggregation (based on ADIOfor Lustre) Topology-aware aggregator mapping on the Tofu
Interconnect with FEFS I/O request throttling among aggregator processes inaccessing OSTsPerformance Impacts of Optimizations (1) HPIO benchmark
 MPI-IO benchmark by using derived data types
 Derived data types generated by the combination of1.
2.
3.

region size
region space (data blank)
region count : the number of regions per MPI process

Performance impacts of Optimizations (2)I/O throttling in write phase of two-phase I/O
The number of I/O requests at the same time (req): 1, 2, 4, 8, 12
 Evaluation of performance improvement ratio relative to the original
implementation• Performance improvements with 2 or 4 I/O requests at the same timeoutperformed other cases with stripe size between 4MiB and 128MiB.

Remarks from my personal experience (1)Monitoring metrics for tracing I/O behavior
Curious to know monitoring metrics for internal Lustre activities orstatistics from a huge number of parameters Providing a light-weight tracing library compatible with OTF would beuseful.

Hoping to get suchtracing informationin Lustre case.

PVFS2 servers MPI processes



MPI_File_write_all()

BMI
Trove(write)

Waiting forclient write
requests

Trove(read)

Tracing Performance of MPI-I/O with PVFS2:
A Case Study of OptimizationY. Tsujita, J. Kunkel. S. Krempel, T. Ludwig, ParCo’09

Remarks from my personal experience (2)The number of service threadsCan we have some QoS management for the available number of threads?
 administrative mode or user application level management

Summary
 MPI-IO performance for Lustre(including FEFS) shouldbe improved to play a big role for data-intensive
parallel applications.
 It would be interesting to have flexible and dynamic
configuration management available by user programs
such as QoS in the available number of service threads
on MDS or OSS.
 R&D in open source community is very welcome forfurther improvements of Lustre functionality.