2023/8 ASE 45thh3-Open-SYS/WaitIO: A System-wide
Heterogeneous Communication Library toCouple Multiple MPI programs
Shinji Sumimoto, Takashi Arakawa, Yoshio Sakaguchi,
Hiroya Matsuba*, Hisashi Yashiro, Toshihiro Hanawa,
Kengo Nakajima
2023/11/29*Achievements as a visiting professor at the University of Tokyo, 2021

2023/11 ASE 45th

22h3-Open-BDEC Innovative Software Platform
for Integration of (S+D+L) on the BDEC
System, such as Wisteria/BDEC-01• “Three” Innovations– New Principles for Numerical
Analysis by Adaptive Precision,
Automatic Tuning & AccuracyVerification– Hierarchical Data Driven Approach(hDDA) based on Machine Learning– Software & Utilities for
Heterogenous Environment, such as
Wisteria/BDEC-01

h3-Open-BDEC
Numerical Alg./LibraryApp. Dev. Framework

Control & Utility

New Principle for
Computations

Simulation + Data +Learning

Integration +Communications+Utilities

Algorithms with HighPerformance, Reliability,
Efficiency

h3-Open-APP: Simulation
Application Developmenth3-Open-SYS
Control & Integration

h3-Open-VER
Verification of Accuracy

h3-Open-DATA: Data
Data Scienceh3-Open-UTIL
Utilities for Large-Scale
Computingh3-Open-AT
Automatic Tuningh3-Open-DDA: Learning
Data Driven Approachh3-Open-MATH

Copyright 2023, Shinji Sumimoto@The University of Tokyo

3Wisteria/BDEC-01• Operation starts on May 14, 2021
• 33.1 PF, 8.38 PB/sec by Fujitsu

The 1st BDEC System (Big Data &Extreme Computing)Platform for Integration of (S+D+L)• ~4.5 MVA with Cooling, ~360m 2

Platform for Integration of (S+D+L)
Big Data & Extreme Computing• 2 Types of Node Groups

Simulation Nodes:Odyssey– Hierarchical, Hybrid, Heterogeneous (h3)– Simulation Nodes: Odyssey• Fujitsu PRIMEHPC FX1000 (A64FX), 25.9 PF– 7,680 nodes (368,640 cores), Tofu-D– General Purpose CPU + HBM– Commercial Version of “Fugaku”

Fujitsu/Arm A64FX
25.9PF, 7.8 PB/s

Shared File
System（SFS）
25.8 PB, 500 GB/s

2.0 TB/s

Data/Learning
Nodes: AquariusIntel Ice Lake + NVIDIA A100

Fast File
System（FFS）
1 PB, 1.0 TB/s

7.20 PF, 578.2 TB/s

– Data/Learning Nodes: Aquarius

800 Gbps

External• Data Analytics & AI/Machine Learning
Resources• Intel Xeon Ice Lake + NVIDIA A100, 7.2PF
External Network– 45 nodes (90x Ice Lake, 360x A100), IB-HDR• Some of the DL nodes are connected to external resources directly• File Systems: SFS (Shared/Large) + FFS (Fast/Small)

ExternalResourcesh3-OpenSYS/WaitIO:
for Heterogeneous Coupling Computing• With the spread of higher-scale computing
systems, HPC applications have become
able to solve more practical problems.
• Example 1: Realizing real-time simulation withvarious kind of sensor data and estimate thefuture.
• Example 2: Improving calculation accuracy byassimilating calculation results and real-timedata.
• Example 3: Using computer simulation results
as machine learning data and doing simulation
using inference computation.

• Fusion of multiple applications is the key
with Simulation, Data processing andmachine Learning

ライフライン事業者（電気、Observation Network
for Earthquake:O(105)）観測網（数万？）Points
ガス、鉄道etc,
震度観測網（約4000点）

強震観測網（約2000点）[c/o Furumura]（例）東京ガス
超高密度地震防災システム（4000点）

Fast Network

JDXnetCase 1Case 2
Case N

Real-Time Data/Simulation Assimilation
Real-Time Update of Underground Model

App. A* Also applicable to full coupling,
multiple applications

2. Send-data extraction fromthe buffer, and data sending

1. Data-packing
into a buffer
4. Data extraction
from the buffer

3. Data-packing after theinterpolation process

App. B• Heterogeneous Coupling Computing by WaitIO
2023/11 ASE 45th

Copyright 2023, Shinji Sumimoto@The University of Tokyo

4

Real Time Heterogeneous Coupling Computing using
WaitIO on Wisteria/BDEC-01
NIED
University’s
Originallydeveloped inERI/U.Tokyo

Output&Movie

ERI

TDX

JDXnetIP Network

Univ.

Local univ.

L2VPN

JMA

SINET5 / JGN

IP Network

Observed Data

Data/Learning
Nodes: Aquarius

FilterVisualizer

Intel Ice Lake + NVIDIA A100

7.20 PF, 578.2 TB/s

h3-Open-SYS/WaitIO-Socket
Obs. Data

External Server
for Observed
Data

2.0 TB/s

Simulation Nodes:
Odyssey
Fujitsu/Arm A64FX
25.9PF, 7.8 PB/s

Seism3D/
OpenSWPC-DAF
Mesh DataUG ModelSimulation and Python Application Couplingby h3-Open-UTIL/MP (h3o-U/MP) + h3-Open-SYS/WaitIO
on Wisteria/BDEC-01

HPC App(Fortran)Analysis/ML
App(Python)

Surrogate
Model
Visualiztion

F<->P adapter

A huge amount ofsimulation data
outputh3o-U/MPh3o-U/MP

Statistics

Coupling

IB-EDR
Odyssey

Aquarius
67API of WaitIO: PB (Parallel Block) == Each Application• Application is able to select communication processes among PBs
WaitIO API

Description

waitio_isend

Non-Blocking Send

waitio_irecv

Non-Blocking Receive

waitio_wait

Termination of waitio_isend/irecv

waitio_init
waitio_get_nprocs

Initialization of WaitIO
Process # for each PB (Parallel Block)

p4

p1

WG
p2

p3

waitio_create_group
Creating communication groups
waitio_create_group_wranks among PB’s
waitio_group_rank

Rank ID in the Group

waitio_group_size

Size of Each Group

waitio_pb_size

Size of the Entire PB

waitio_pb_rank

Rank ID of the Entire PB

2023/11 ASE 45th

Copyright 2023, Shinji Sumimoto@The University of Tokyo

7

WaitIO-MPI Conversion Library• WaitIO does not have MPI Datatype• WaitIO-MPI Conversion Library was developed to convert MPI
program to WaitIO

WaitIO-MPI API (2022/9/1)
description‘
WaitIO MPI_Isend
waitio_mpi_isend
waitio_mpi_irecv
WaitIO MPI_Irecv
WaitIO MPI_Reduce
waitio_mpi_reduce
WaitIO MPI_Bcast
waitio_mpi_bcast
WaitIO MPI_Allreduce
waitio_mpi_allreduce
WaitIO MPI_Waitall
waitio_mpi_waitall
WaitIO Initialization(all ranks)
waitio_create_universe
waitio_create_universe_pbhead WaitIO Initialization(rank0 of each PB)
WaitIO MPI_Gather
waitio_mpi_gather
waitio_mpi_algather
WaitIO MPI_Algather
WaitIO MPI_Scatter
waitio_mpi_scatter
WaitIO MPI_Scatterv
waitio_mpi_scatterv
WaitIO MPI_Gatherv
waitio_mpi_gatherv
WaitIO MPI_Barrier
waitio_mpi_barrier
WaitIO MPI_Tyep_size
waitio_mpi_type_size2023/11 ASE 45th

Copyright 2023, Shinji Sumimoto@The University of Tokyo8Existing File Coupling Application Program
Conversion by WaitIO Easily• Converting a Sample Coupling Program using File Coupling toWaitIO Coupling• Toy Programs of 3D unsteady-state heat transfer problems with the finite
element method (FEM) using iterative linear solvers• Converting Three Toy Program Coupling from file based to WaitIO Based
Code-1

Code-2

Code-3

Computing Point source
calculation

Unsteady threedimensional heat
conduction equation byFEM (MPI)

Unsteady threedimensional heat
conduction equation byFEM (MPI)

Input

s1.dats2.dat, s1.outs3.dat, s1.out, s2.out

Outputs1.out, t1.lst

s2.out, t2.lst

S3.out, t3.out
Code-3: Reading Code-2 Output
Forced temperature fixed
conditions in the plane of Z= Zmax

2023/11 ASE 45th

Copyright 2023, Shinji Sumimoto@The University of Tokyo

9

Code-1 Program Source: File Based• Code-1: Original Code Using File Coupling

2023/11 ASE 45th

Copyright 2023, Shinji Sumimoto@The University of Tokyo

10

Code-1 Program Source: WaitIO Based• Converting Open/Read-Write to isend-irecv/wait• MPI Non-blocking Send/Recv Style Conversion

2023/11 ASE 45th

Copyright 2023, Shinji Sumimoto@The University of Tokyo

11

The pHEAT-3D Application Conversion Example• Translates pHEAT-3D from Fortran+MPI to WaitIO-MPIConversion Library• WaitIO-MPI Conversion code conversion mechanically convertible
include 'mpif.h'include 'waitio_mpif.h'
integer(kind=kint ), dimension(:,:), save,allocatable :: req1
integer, save :: NFLAG
data NFLAG/0/!C!C-- INIT.
if (allocated(sta1)) deallocate (sta1)
if (allocated(req1)) deallocate (req1)
allocate (sta1(WAITIO_STATUS_SIZE,2*NEIBPETOT+4))
allocate (req1(WAITIO_REQUEST_SIZE,2*(NEIBPETOT+4)))!C!C-- SEND
do neib= 1, NEIBPETOT
istart= STACK_EXPORT(neib-1)
inum = STACK_EXPORT(neib ) - istart!$omp parallel do private (k,ii)
do k= istart+1, istart+inum
ii= NOD_EXPORT(k)WS(k)= X(ii)enddo
call WAITIO_MPI_Isend (WS(istart+1), inum,&WAITIO_MPI_DOUBLE_PRECISION,&NEIBPE(neib), 0, WAITIO_SOLVER_COMM, req1(1,neib), ierr)enddo

2023/11 ASE 45th

!C-- RECV
do neib= 1, NEIBPETOT
istart= STACK_IMPORT(neib-1)
inum = STACK_IMPORT(neib ) - istartcall WAITIO_MPI_Irecv (X(istart+N0+1),inum,&WAITIO_MPI_DOUBLE_PRECISION,&NEIBPE(neib), 0, WAITIO_SOLVER_COMM,&req1(1,neib+NEIBPETOT), ierr)enddo!Ccall WAITIO_MPI_Waitall (2*NEIBPETOT, req1, sta1, ierr)
end subroutine SOLVER_SEND_RECV
end module
solver_SR!$omp parallel do private(i) reduction(+: RHO0)
do i= 1, N
RHO0= RHO0 + WW(i,R)*WW(i,Z)
enddo

call WAITIO_MPI_Allreduce (RHO0, RHO, 1,&WAITIO_MPI_DOUBLE_PRECISION,&WAITIO_MPI_SUM, WAITIO_SOLVER_COMM, ierr)

Copyright 2023, Shinji Sumimoto@The University of Tokyo

12

13

h3-Open-UTIL/MP +h3-Open-SYS/WaitIO-Socket
Operation Started in June 2022

h3open modules
Jcup modules

Jcup modules

jcup_mpi_lib.f90

jcup_mpi_lib.f90

Aquarius

Fortran APP(NICAM)

Python APP(PyTorch)

h3open modules

IB-EDR

h3opp.pyh3open_py.f90
h3open modules

Jcup

h3opp.pyh3open_py.f90

h3-Open-UTIL/MP

Python APP(PyTorch)

h3open modules

Jcuph3-Open-UTIL/MP

Fortran APP(NICAM)

OdysseyJcup modules

Jcup modules

jcup_mpi_lib.f90

jcup_mpi_lib.f90

MPI wrapper

MPIMPI

WaitIOMPI

MPI+WaitIOMay
2021: Single MPI Environment
2023/11 ASE 45th

June 2022: Coupler＋WaitIO
13

14

Platform for Integration of (S+D+L)
Big Data & Extreme Computing

How to run the workloads• Total Number of Nodes– Odyssey: 7,680 nodes: not so crowded– Aquarius: 45 nodes, 360 GPUs, very
crowded• One node of Aquarius is reserved
for this type of workload on theintegration of (S+D+L)• 2 separate jobs (Odyssey, Aquarius)
should be submitted• If both jobs “grab” resources,execution starts.
• More flexible (& complicated) policy
neededSimulation Nodes:
Odyssey
Fujitsu/Arm A64FX
25.9PF, 7.8 PB/s

Shared File
System（SFS）
25.8 PB, 500 GB/s

2.0 TB/s

Data/Learning
Nodes: AquariusIntel Ice Lake + NVIDIA A100

Fast File
System（FFS）
1 PB, 1.0 TB/s

7.20 PF, 578.2 TB/s

800 Gbps

ExternalResources

External
Resources
External Network

HPC App(Fortran)

Analysis/MLApp(Python)

Surrogate
Model
Visualiztion

F<->P adapter

A huge amount ofsimulation dataoutput

h3o-U/MP

h3o-U/MP

Statistics

Coupling

IB-EDR
Odyssey

Aquarius

Job Co-execution between Different Batch
Processing Systems• We developed:• WaitIO Hostname Server:

Platform for Integration of (S+D+L)
Big Data & Extreme Computing• Exchanging host information

Simulation Nodes:
Odyssey• Coupled Job Execution Service:

Fujitsu/Arm A64FX
25.9PF, 7.8 PB/s

• Watching job scheduler status and dispatching
coupling jobs

Shared FileSystem（SFS）• Coupled Job Execution Sequences:• Classification between Regular Execution
Jobs and Coupled Execution Jobs
• Do Simultaneous Execution: When
resources of the multiple systems areavailable• WaitIO-Socket execution environment
settings
2023/11 ASE 45thTCS-o

25.8 PB, 500 GB/s

2.0 TB/s

Fast File
SystemData/Learning
Nodes: AquariusIntel Ice Lake + NVIDIA A100

7.20 PF, 578.2 TB/s

（FFS）
1 PB, 1.0 TB/s

TCS-a800 Gbps

External
Resources

Copyright 2023, Shinji Sumimoto@The University of Tokyo

External
Resources
External Network

WaitIO HostnameServer

Coupled Job
Execution Service15

WaitIO-Socket, WaitIO-File vs. WaitIO-Hybrid PingPong
Bandwidth Odyssey 2 node (Homogeneous)• WaitIO-Hybrid：Achieved 1.1GB/s• Rendezvous Protocol is accelerated by WaitIO-Socket
Odyssey 2 Node Communication Performance Scaling:

MB/s

1,200

1079.55

1,000
800
600
400
2000
1.0E+00

2023/11 ASE 45th

1.0E+01

1.0E+02

1.0E+03

1.0E+04

1.0E+05

1.0E+06

1.0E+07

WaitIOHybrid(1proc/node)

WaitIO-Hybrid(2proc/node)

WaitIO-Hybrid(4proc/node)

WaitIO-Hybrid(8proc/node)

WaitIO-Hybrid(16proc/node)

WaitIO-Socket(1proc/node)

WaitIO-Socket(16proc/node)

WaitIO-File(1proc/node)

WaitIO-File(16proc/node)

Copyright 2023, Shinji Sumimoto@The University of Tokyo

1.0E+08

Bytes

16h3-Open-UTIL/MP+h3-Open-SYS/WaitIO
Application Performance Evaluation• NICAM-AI(ADA): Performance Evaluation of Coupling Computing• Total air density, Internal energy, Density of water vapor• Framework : PyTorch, Method : Three-Layer MLP• Resolution : horizontal : 10240, vertical : 78
HPC App(Fortran)

Analysis/ML
App(Python)

Surrogate
Model
Visualiztion

F<->P adapter

A huge amount ofsimulation dataoutput

h3o-U/MP

h3o-U/MPStatistics

Coupling

ML Processing Model for Climate Simulation
NICAM＋ADA Execution Image

2023/11 ASE 45th

Copyright 2023, Shinji Sumimoto@The University of Tokyo17

WaitIO Performance: NICAM+ADA• Comparison of Socket, File, Hybrid• WaitIO-Hybrid was best performance
Sec.
700
Seconds

SocketFile

NICAM-o ADA-o NICAM-a ADA-a
320.3
370.7

Hybrid

306.5

467.9
576.6
452.1

293.9
247.3
213.1

600

350.1

500

303.8

400

268.4

300

NICAM+ADA Application Performance
Odyssey + Odyssey
Homogeneous

Odyssey + Aquarius
Heterogeneous

2001000NICAM-oADA-oSocket2023/11 ASE 45th

Copyright 2023, Shinji Sumimoto@The University of Tokyo

NICAM-aFileADA-aHybrid18

Summary of WaitIO(h3-OpenSYS/WaitIO)• High performance System-wide communication for:• Coupling multiple MPI applications
among multiple heterogeneous systems

Platform for Integration of (S+D+L)
Big Data & Extreme Computing• No extra software needed• Works among systems withPOSIX Socket communication and Shared File

Simulation Nodes:
Odyssey
Fujitsu/Arm A64FX
25.9PF, 7.8 PB/s

• Existing related work needs same software stack among wholesystem nodes, so interoperability is limited

Shared FileSystem• Existing MPI and compiler can be used（SFS）• Fujitsu MPI, Intel MPI and Open MPI for GPU

25.8 PB, 500 GB/s

• Future Work

2.0 TB/s

Fast File
SystemData/Learning
Nodes: Aquarius（FFS）
1 PB, 1.0 TB/sIntel Ice Lake + NVIDIA A100

7.20 PF, 578.2 TB/s

• WaitIO-Verbs for InfiniBand, RoCE, Slingshot(Now Working)• Real Application Porting: Several Projects are on-going800 Gbps

External
Resources

External
Resources
External Network• Acknowledgement• This work is supported by “JSPS Grant-in-Aid for Scientific Research(S) (19H05662)”,and by "Joint Usage/Research Center forInterdisciplinary Large-scale Information Infrastructures(jh210022-MDH)".
2023/11 ASE 45th

Copyright 2023, Shinji Sumimoto@The University of Tokyo

19

Questions?

2023/11 ASE 45th

Copyright 2023, Shinji Sumimoto@The University of Tokyo

20