Performance improvement of MODYLAS using
Remote Direct Memory Access on the K computer
1112

Masahiro Nakao, Hitoshi Murai, Mitsuhisa Sato, Yoshimichi Andoh, Susumu Okazaki(1. RIKEN Center for Computational Science, 2. Nagoya University)

What is MODYLAS ? Summary

MOlecular DYnamics software for LArge System

In order to improve the performance of MODYLAS, thisresearch replaces MPI communication with Remote Direct
Memory Access (RDMA) on the K computer

MODYLAS utilizes the fast multipole
method (FMM) for the calculation ofthe electrostatic interactions

Since the K computer provides the extended RDMA
interface for RDMA operations, we implement a library touse the interface from MODYLAS easily

MODYLAS is executed onlarge-scale supercomputers such as
the K computer (right ﬁgure)
Our preliminary evaluation indicates
the time required for MPI communication is limited by its latency

SPARC64 VIIIfx 2GHz,
DDR3 SDRAM 16GB,
Tofu interconnect 5GB/s

ApproachModiﬁed code of MODYLAS
We implement a library to use the extended RDMA interface
from MODYLAS easily

Replacement MPI with RDMA
The K computer provides users with the extended RDMAinterface so that they can issue RDMA operations (Put/Get)with low latency

Latency (microsecond)This graph shows a comparison of latency between MPI andRDMA (Put) on the K computer using ping-pong benchmark
10 3

MPI
RDMA (Put)

10 2

10 1

RDMA (Put) isalways better

10 0
864

512

4K

32K

256K

2M

Transfer data size (Byte)

Evaluation

integer(4),allocatable,dimension(:) :: icbufp
allocate(ircbufp(s))
Register array to#ifdef RDMA
use RDMA
call rdma_register_addr(ircbufp, s*4)
#endif:#ifdef RDMA
Get informationinteger(8),pointer :: ircbufp_raddr(:)
of remote arraystype(c_ptr) :: ircbufp_cptr
ircbufp_cptr = rdma_get_raddr(ircbufp)
call c_f_pointer(ircbufp_cptr, fptr=ircbufp_raddr, shape=[nprocs])
call rdma_put_post(ipz_pdest, ircbufp_raddr(ipz_pdest+1), ...)
call rdma_wait(ipz_psrc)#else
Perfom RDMA PUTcall mpi_irecv(ircbufp, ..., ipz_psrc, ...)
communication
call mpi_isend(icbufp, ..., ipz_pdest, ...)
call mpi_waitall(2, ...)
#endifCalculation time using MPI and RDMA

Communication time using MPI and RDMA
RDMA communication time is 29-42% less than MPI
communication time on the data set with three FMM levels
Most transfer data sizes are less than 32K bytes, which is a
suﬃcient size to demonstrate the superiority of RDMA

Time (microsecond) per step

As a result of measuring the performance of MODYLAS, theRDMA communication time is 29-42% less than theMPI communication time

This table shows the total calculation time including
communication time per step
Although the eﬃciency has increased by a factor of 2.91~4.68% overall, this will further increase for calculations withstrong scaling with tuned code for hotspot calculations

800700MPIRDMA (Put)

600
500

Future Work400

To make MODYLAS available for reducing communication
times in various computing environments, we will utilize
coarray features of the Fortran standard

300
200
10008163264

Number of Processes

128

256

Since the coarray features provide users with one-sided
communication, and its implementation may use RDMA thateach machine has2