高性能計算研究室
RDMAwithDouble
Bufferingfor
Advanced Software Laboratory
High Performance Computing Laboratory
Adjacent Communication

先進ソフトウェア研究室

Kota Yoshimoto, Akihiro Fujii, Teruo Tanaka
Kogakuin UniversityIntroduction
In large-scale scientific computing programs, parallelization by MPI communication is generally used. MPI is convenient because it can be
executed on many computers. However, interprocess communication often becomes a bottleneck in highly parallel computers [1].
In this research, we measured and compared the performance of MPI, RDMA, and RDMA with double buffering for adjacent communication.

Adjacent Communication➢it communicates with the front two processes and the back two processes.

➢ MPI_Isend/Irecv➢ RDMA with Double Buffering•••

It is a communication method generally
used for adjacent communication of MPI.
MPI_Isend / Irecv is two-way communication.

➢ Communication setting

It does not require synchronization byusing two buffers for communication.

Process1
communicateswith process 7,process 8, process
2, and process 3.

➢ RDMA••

RDMA communication can read and write data
without the intervention of the program of thedestination node by using a dedicated memory.
RDMA is one-sided communication.

Fig.3 Adjacent Communication
with 4 neighboring process case.

In RDMA
communication, it
is necessary tosynchronize so thatthe data will not be
overwritten while
it is being used.

Fig.1 RDMA

Data to be sent is stored
in a and b. Received datais stored in c and d.

Fig.2 RDMA with Double Buffering:
When Buffer1 reception is not finish, Calculation
process cannot start. Therefore, communication
timing gap will less than 2.

Fig.4 Vector image on processes
from rank0 to rank2

Numerical experiment
In this experiment, we used a supercomputer Flow at Nagoya
University. It is equipped with a system called FX1000 [2].

Experiment Condition• The number of processes is 128.
• Double precision data is 160.

➢ Fig.5 shows communication time when the total
number of processes is changed.
➢ Fig.6 shows communication time and improvement
rate when the number of adjacent processes ischanged.

Fig.6.1 Communication time of adjacent communication
when the number of adjacent communicationadjacent processes is changed.

The communication time increases proportionallywhen the number of adjacent processes is changed.
This is because changing the number of adjacentprocesses increases the amount of communicationper process.

RDMA with double buffering was confirmed toimprove adjacent communication time by up to30% compared to MPI.
Fig.5 Communication time of adjacent communication when thetotal number of processes is changed by adjacent communication:
Experiment Condition• The number of adjacent processes is 2
• Double precision data is 160.
The communication time does not change much when the total
number of processes is changed. This is because the amount ofcommunication per process does not change even if the total
number of processes is increased.

Fig.6.2 Communication time ratio when the MPI time is 1
when the number of adjacent processes is changed inadjacent communication.

Conclusion
MPI, RDMA, and RDMA with double buffering were evaluated using simple adjacent communication with various number ofneighboring processes and different total process numbers. It showed that RDMA was faster than MPI as the number of adjacentprocesses increased. In addition, RDMA with double buffering reduced synchronization and was the most efficient communication
method of the three by up to 30%.
Further
researchon
thewaytoapply
thistechniquetoapplication
programs
usingadjacentcommunication.
Acknowledgments
This work is supported by “Joint Usage/Research Center for Interdisciplinary Large-scale Information Infrastructures” and “HighPerformance Computing Infrastructure” in Japan (Project ID: jh210026-NAH).
Reference[1] Kanamori Issaku, et al., Acceleration of communication with low latency uTofu interface in LQCD application, IPSJ SIGHPC
report, Vol.2020-HPC-177 No.22 (2020).
[2] Supercomputer “Flow Type I subsystem”, Nagoya Univerisyty. <https://icts.nagoya-u.ac.jp/ja/sc/>