Correctness Check for XMP in YML--- Programming in multi SPMD Programming Model
Miwako Tsuji

2019.11.05 7th XMP Workshop

XMP+YML and FP3C project FP3C: Framework and Programming for Post Petascale Computing
 a collaborative project between Japan and France September. 2010 – March. 2014

 Various research fields and their integration
 Programming model and programming language design
 Runtime libraries
 Accelerator
 Algorithm and mathematical libraries
 etc…

2013? @ Akihabara

MYX Project Consortium MUST Correctness Checking for YML and XMP Programs.
 International collaboration among Germany (DFG), Japan (JST), and France (ANR).
 Part of the Priority Programme "Software for Exascale Computing" (SPPEXA) in German.

• Partner from Germany (project coordinator)-RWTH Aachen, IT Center and Institute for High
Performance Computing-Prof. Matthias S. Mueller, Joachim Protze, ChristianTerboven• Partner from Japan-University of Tsukuba, Center for Computational Sciences,
and Center for Computational Science, RIKEN-Prof. Taisuke Boku, Hitoshi Murai, Miwako Tsuji• Partner from France-Maison de la Simulation-Prof. Serge Petiton. Prof. Nahid Emad, Thomas Dufaud3

Overview of MYX
 Runtime Correctness check for multi SPMD (mSPMD) programming model MUST (Germany) YML (France)
Trilateral collaboration for scalable and productive computation
 XMP (Japan)
Correctness checking for
XMP in XMP+YML

YML:

MUST:
correctness checking
tool for MPI

workflow environment

XMP provides a tool interface
XMPT for analyses

YML orchestrates
multiple applications

Must checks Correctness
for XMP using XMPTXMP parallelizes YML tasks
XMP+YML=
Hieratical and Scalable
Programming ModelXMP:PGAS based parallel
programming language

MUST: Runtime Correctness Check for MPI
 Must is a runtime correctness check library developed by the RWTH Aachen Detects errors in MPI (and OpenMP) and reports users
 Scalable
Application processes
Softwarestack：MUST uses PnMPI library
to intercept MPI function calls, and analyze them
MUST: MPIstandards, rule for analyses
GTI (Generic Tool Infrastructure)

OMPT

PnMPI

MUST
Process

application
Event
PnMPI
Local Analysis

GTI

GTI

MPI library

Non-localAnalysis

Overview of MUSTMemory

MemoryMemoryMemoryMulti SPMD (mSPMD) Programming Model
Heterogeneous
NodeCPUCPU

cpucpucpucpuNUMA Nodehierarchical systems
 Hierarchical systems
 A node may consist of many general cores and accelerator cores
 NUMA topology in a node
 A group of nodes tightly connected, Network locality
 A system consists of groups of nodes / a cluster of clusters
 Multi-programming methodologies across multi-architectural levels
 Software had been developed to execute applications based on this programming modelMulti SPMD (mSPMD) Programming Model

MPI
XMP

YMLMemory

hierarchical systems

Node

Node

Node

Node

Node

Node

<task 2>

<task 3><task 5><task 9><task 10><task 4><task 6>

<task 7><task 8>CPUCPU

cpucpucpucpuSharedMemoryMemoryNUMA Node

UNIVERSITE DE VERSAILLESSAINT QUENTIN EN YVELINES<task 1>

StarPU

MemoryWorkflow

Distributed
Parallel

Heterogeneous
Node

Memorymulti-programming methodologies
across multi-architectural levels

OpenMP
OpenACC
StarPU, …

 Hierarchical systems
 A node may consist of many general cores and accelerator cores
 NUMA topology in a node
 A group of nodes tightly connected, Network locality
 A system consists of groups of nodes / a cluster of clusters

Multi SPMD (mSPMD) Programming Model

MPI
XMP

YML

Memoryhierarchical systems

Node

Node

Node

Node

NodeNode<task 2>

<task 3><task 5><task 9><task 10><task 4><task 6>

<task 7><task 8>CPUCPU

cpucpucpucpuSharedMemoryMemoryNUMA Node

UNIVERSITE DE VERSAILLESSAINT QUENTIN EN YVELINES<task 1>

StarPU

MemoryWorkflow

Distributed
Parallel

Heterogeneous
Node

Memorymulti-programming methodologies
across multi-architectural levels

OpenMP
OpenACC
StarPU, …

 Hierarchical systems
 A node may consist of many general cores and accelerator cores NUMA topology in a node
 A group of nodes tightly connected, Network locality
 A system consists of groups of nodes / a cluster of clusters

XcalableMP (XMP) Directive based parallel programming
language Data distribution and work mapping can be
declared by XMP directives
 XMP Compiler
 Source-to-source compiler
 C+XMP ⇒ C+XMP-runtime library call
 The XMP runtime library uses MPI in its
communication layer

int B[12];#pragma xmp nodes p(4)Data Mapping#pragma xmp template t(0:11)#pragma xmp distribute t(block) ont p#pragma xmp align B[i] with t(i)a one-dimensional block-distributed array B[]
distributed over four nodes
0 1 2 3 4 5 6 7 8 9 10 11
Node1
Node2

Node3
Node4

Data can be distributed over different
processes of a task automatically#pragma xmp loop (i) on t(i)for(i=0; i<12; i++){B[i] = B[i]*2;}

Work MappingMulti SPMD (mSPMD) Programming Model
MemoryMemoryMemoryWorkflow

NUMA Nodehierarchical systems<task 1>

Node

Node

Node

Node

NodeNode<task 2><task 3><task 5><task 9><task 10><task 4><task 6>OpenMP
OpenACC
StarPU, …

<task 7><task 8>••CPUCPU

cpucpucpucpuSharedMemoryMemoryDistributed
Parallel

Heterogeneous
Nodeintroduce “parallelism” into tasks by XMP“heavy” task can be executed in parallel

Multi SPMD (mSPMD) Programming Model
MemoryMemoryMemoryWorkflow

CPUCPU

cpucpucpucpuSharedMemoryMemoryDistributed
Parallel

Heterogeneous
Node

NUMA Nodehierarchical systems<task 1>

Node

Node

Node

Node

Node

Node

<task 2>

<task 3><task 5><task 9><task 10><task 4><task 6>OpenMP
OpenACC
StarPU, …

<task 7><task 8>• divide a large parallel program into some- coarse
sub-programs
to avoidgrained
the costtasks
of in a workflow- moderate
size SPMD programs
communicationin large systems

Multi SPMD (mSPMD) Programming Model
MemoryMemoryMemoryWorkflow

NUMA Nodehierarchical systems<task 1>

Node

Node

Node

Node

Node

Node

<task 2>

<task 3><task 5><task 9><task 10><task 4><task 6>OpenMP
OpenACC
StarPU, …

<task 7><task 8>CPUCPU

cpucpucpucpuSharedMemoryMemoryDistributed
Parallel

Heterogeneous
Node• Compose complex application by
combining parallel appreciations andlibraries

Multi SPMD (mSPMD) Programming Model
MemoryMemoryMemoryWorkflow

CPUCPU

cpucpucpucpuSharedMemoryMemoryDistributed
Parallel

Heterogeneous
Node

NUMA Nodehierarchical systems<task 1>

Node

Node

Node

Node

Node

Node

<task 2>

<task 3><task 5><task 9><task 10><task 4><task 6>

<task 7><task 8>Fault
detection
OpenMPand recovery
OpenACC
StarPU, …

Multi SPMD (mSPMD) Programming Model
MemoryMemoryMemoryWorkflow

CPUCPU

cpucpucpucpuSharedMemoryMemoryDistributed
Parallel

Heterogeneous
Node

NUMA Nodehierarchical systems<task 1>

Node

Node

Node

Node

Node

Node

<task 2>

<task 3><task 5><task 9><task 10><task 4><task 6>

<task 7><task 8>OpenMP
Deadlock inside
OpenACC
each task?
StarPU, …

The fault detection can notdetect deadlocks in MPI
programs

Multi SPMD (mSPMD) Programming Model Scalability
 by combining different parallel programming paradigm across different architectural level
 Reliability
 fault tolerant futures have been supported
 Productivity
 correctness checking by MUST library

MUST+YML+XMP (MYX)

invocation

communication

Overview of execution of mSPMD programming model
Node0

Node1

mpirunMPI_Comm_spawn

Node2

Node3

Node4

MPI_Comm_spawn
MPI_Send(“task2”, ...

remote program1

remote program2

yml_scheduler &OmniRPC-MPI library<task2><task1>#pragma xmp loop on t(i)#pragma xmp task on p(3)MPI_Comm_spawn
MPI_Send(“task3”, ...
MPI_Barrier(…);(wait)(wait)
remoteremote<task3>
program3
program4
MPI_Send(…
<task4>
MPI_Allreduce(…
<task5>

MUST+YML+XMP (MYX)

invocation

communication

Target of correctness check in execution of mSPMD programming model
Node0

Node1

mpirunMPI_Comm_spawn

Node2
Node3SPMD tasks Node4• Check user defined(XMP, MPI) by MUST
• Ignore the communication for workflow controls in the
MPI_Comm_spawn
middleware
MPI_Send(“task2”, ...

remote program1

remote program2

yml_scheduler &OmniRPC-MPI library<task2><task1>#pragma xmp loop on t(i)#pragma xmp task on p(3)MPI_Comm_spawn
MPI_Send(“task3”, ...
MPI_Barrier(…);(wait)(wait)
remoteremote<task3>
program3
program4
MPI_Send(…
<task4>
MPI_Allreduce(…
<task5>
Apply the correctness check by MUST for each task

XMPT Tool Interface
 ... is a generic tool API of XMP.
 Its basic idea is inspired by OMPT.
 event- and callback-based (Planned) targets:
 MYX (SPPEXA project by RWTH Aachen, UVSQ, and R-CCS)
 Extrae @ BSC
 Score-P / Scalasca @ JSC
 etc.

Oct. 21, 2019

SPPEXA Final Symposium

21Basic Design of XMPTCallbacks are registeredthrough xmpt_set_callback.

◼ At initialization

void xmpt_set_callback(...);

Provided by an XMP compiler.

Provided by tools

void xmp_init(){xmpt_initialize(...);...
}

void xmpt_initialize(...){
xmpt_set_callback(XMPT_BCAST_BEGIN, myx_bcast_begin);xmpt_set_callback(XMPT_BCAST_END, myx_bcast_end);...
}

xmp_init invokes
xmpt_initialize.

◼ At each event

void xmpt_initialize(...) __attribute__((weak));

The registered callbacks are invoked.

void xmp_bcast(...){(*xmpt_bcast_begin)(...);
xmp_bcast_body(...);(*xmpt_bcast_end)(...);}

Mar. 18, 2019void
myx_bcast_begin(...);
void
myx_bcast_end(...);

CRIStAL laboratory, Lille

22

MUST+YML+XMP (MYX): Implementation
 MUST+MPI / MUST+XMP : to check a single SPMD program mustrun –np n application.exe
 prepare a dedicated dynamic library for the application.exe, set the environmental variables
 mpirun –np (n+1) application.exe: 1 process should be kept for the MUST analysis
 MUST+YML+MPI/XMP: to check multiple SPMD program Instead of mustrun (mpirun), MPI_Comm_spawn is used to invoke remote SPMD programs inmSPMD
 extend the middleware of workflow scheduler and the remote program generator in mSPMD
 MPI_* functions in the workflow control are replaced with PMPI_* functions
 MPI_Comm_spwan(“prog”, n, …) ➝ PMPI_Comm_spwan(“prog”, n+1, …)
 preparation steps performed within the mustrun script before mpirun should be performed
before starting a workflow
 set the environmental variables required by MUST manually (Originally, they are set by themustrun scprit) prepare a dedicated dynamic library to analyze each remote programExperiments on Oakforest-PACS
 Compare the behavior of workflow applications w/ and w/o error
 Evaluate the overhead to apply MUST for tasks in a mSPMD application
 Oakforest-PACS (OFP): supercomputer installed in Kashiwa, operated by U. Tokyo and U.
Tsukuba 8208 KNL nodes, Connected via Intel Omni Path
 Compiler intel/2018.1.163
 MPI impi/2018.1.163
 30 processes (flat-MPI) for each task, 1, 2, 4, 8, 16, 32 tasks in each application, all tasks arerun simultaneouslyTest codes
Allreduce
for(i=0; i<100; i++){MPI_Allreduce(buf, rbuf, 1, MPI_LONG, MPI_SUM, MPI_COMM_WORLD);
usleep(100000);}
Allreduce: Type conflictfor(i=0; i<100; i++){if(myrank==0)
MPI_Allreduce(buf, rbuf, 1, MPI_INTEGER, MPI_SUM,MPI_COMM_WORLD);
else
MPI_Allreduce(buf, rbuf, 1, MPI_INT, MPI_SUM, MPI_COMM_WORLD);
usleep(100000);}

Test codes (cont.
Allreduce: Operation conflictfor(i=0; i<100; i++){if(myrank==0)
MPI_Allreduce(buf, rbuf, 1, MPI_LONG, MPI_MAX, MPI_COMM_WORLD);
else
MPI_Allreduce(buf, rbuf, 1, MPI_LONG, MPI_MIN, MPI_COMM_WORLD);
usleep(100000);}
Allreduce: Size conflictfor(i=0; i<100; i++){if(myrank==0)
MPI_Allreduce(buf, rbuf, 1, MPI_LONG, MPI_SUM, MPI_COMM_WORLD);
else
MPI_Allreduce(buf, rbuf, 2, MPI_LONG, MPI_SUM, MPI_COMM_WORLD);
usleep(100000);}

Test codes (cont.
Pingpong
for(i=0; i<100; i++){if(myrank%2==0) MPI_Send(buf, 1, MPI_LONG, dest, tag, MPI_COMM_WORLD);else
MPI_Recv(buf, 1, MPI_LONG, src , tag, MPI_COMM_WORLD, ..);
usleep(100000);if(myrank%2==0) MPI_Recv(buf, 1, MPI_LONG, src , tag, MPI_COMM_WORLD,..);else
MPI_Send(buf, 1, MPI_LONG, dest, tag, MPI_COMM_WORLD);
usleep(100000);}
Pingpong, Type conflictfor(i=0; i<100; i++){if(myrank%2==0) MPI_Send(buf, 1, MPI_UNSINGED_LONG, dest, tag, MPI_COMM_WORLD);else
MPI_Recv(buf, 1, MPI_LONG, src , tag, MPI_COMM_WORLD, ..);
usleep(100000);if(myrank%2==0) MPI_Recv(buf, 1, MPI_LONG, src , tag, MPI_COMM_WORLD,..);else
MPI_Send(buf, 1, MPI_LONG, dest, tag, MPI_COMM_WORLD);
usleep(100000);}

Result (1) Status
w/ MUST

w/o MUST

completed?

reported?

completed?

reported?

allreduce w/oerror

completed-completed-allreduce typeconflict

completederror reportcompletednoallreduce
operation conflictcompletederror reportcompletednoallreduce size
conflict

failederror report

failedsimple error
reportpingpong w/oerror

completed-completed-pingpong typeconflict

completederror reportcompletednoResult (2) Example of error report from MUST

Result (3) Overhead: MPI_Allreduce(sec)(Allreduce + 1-sec sleep )x100(sec)(Allreduce + 0.01-sec sleep )x10000

200

100

80

150

60

100

40

50

2000
124816

NoError-NoMUST

NoError-MUST

Error-NoMUST

Error-MUST

32

tasks

124816

NoError-NoMUST

NoError-MUST

Error-NoMUST

Error-MUST The overheads depend on the frequency of the communication
 The overhead is ignorable if we don’t perform communication very intensively
 Some overheads even if there is no error if we call MPI_allreduce 100 times per second32tasks

Result (3) Overhead: pingpong(sec)(Send + 1-sec sleep(sec)Recv + 1-sec sleep )x50
200

200

150

150

100

100

50

50

00

1

24816

NoError-NoMUST

NoError-MUST

Error-NoMUST

Error-MUST

32

tasks

(Send + 0.01-sec sleep
Recv + 0.01-sec sleep )x5000

124816

NoError-NoMUSTNoError-MUST

Error-NoMUST

Error-MUST No overhead if there is no error The overhead to record errors in the point-to-point communications is large (even when 1
point-to-point communication per second) due to the complexity of MPI function calldependencies

32

tasks

Experiments for XMP-tasks: Test codes
Uncorrect: reduction out of nodes#pragma xmp task on nodes(1){#pragma xmp reduction (+:sum) on nodes(3)}p(1)
task on nodes(1)reduction

on nodes(3)

p(2)

p(3)

p(4)

?

Experiments for XMP-tasks: Results

Conclusion
 MYX: an international collaborative project for higher productivity in exascale computing. Runtime
correctness check by MUST for multi SPMD Programming Model by YML+XMP
 MUST is a correctness checking tool.
 YML is a workflow language (to be presented by Miwako) XMP is a directive-based PGAS extension for Fortran & C supporting the global- and local-viewprogramming.
 XMP+MUST
 XMP provides an interfere, XMPT, for performance tools
 MUST uses the XMPT and check the correctness of XMP
 XMP+YML
 Tasks written in XMP of a workflow managed by YML
 MUST+YML+XMP
 The task generator and middleware in mSPMD have been extended⇒ Scalable, reliable programming model with high productively
Scalable : Combination of multiple-SPMDs by YML and XMP
Reliable : Fault-detection and recovery are supported
High Productively : XMP, YML are easier than C+MPI
MUST and XMPT provide a debug tool for SPMD