PCCC HPCオープンソースソフトウェア普及部会ワークショップ「高性能クラスタ・プログラミング最前線」 4月15日 2022年

PGAS並列プログラミング言語 XcalableMPの開発進捗・状況

佐藤 三久
Team Leader, Programming Environment Research Team, RIKEN R-CCS
Professor (Cooperative Graduate School Program), University of Tsukuba

XcalableMPについて

これまで、PCCC XcableMP言語規格部会で仕様を議論し、理研・筑波大で開発を進めてきた。
Omni XcalableMP コンパイラ https://omni-compiler.org/
 https://github.com/omni-compiler/omni-compiler
 XcalableMP 1.x 仕様は、データ並列＋Coarrayをベースしたもの
 MPIのruntimeを使うので、MPIが使えるクラスタならば実行可能


富岳で利用可能 （フラグシップ2020の一部として開発・評価）
XcalableMP 1.x 仕様
 Tofu-Dを用いる実装
XcalableMP(XMP) http://www.xcalablemp.org
 Language Features
What’s XcalableMP (XMP for short)?
 Directive-based language extensions for Fortran and C for A PGAS programming model and language for
PGAS model
distributed memory , proposed by XMP Spec WG
 Global view programming with global-view distributed data
 XMP Spec WG is a special interest group to design
structures for data parallelism
and draft the specification of XcalableMP language. It
is now organized under PC Cluster Consortium,
 SPMD execution model as MPIJapan. Mainly active in Japan, but open for everybody.
 pragmas for data distribution of global array.
 Work mapping constructs to map works and iteration with
 Project status (as of June 2019)affinity to data explicitly.
 XMP Spec Version 1.4 is available at XMP site.
 Rich communication and sync directives such as “gmove” andnew features: mixed OpenMP and OpenACC ,“shadow”.
libraries for collective communications.
 Many concepts are inherited from HPF
 Reference implementation by U. Tsukuba and Co-array feature of CAF is adopted as a part of the language
Riken AICS: Version 1.3.1 (C and Fortran90) isavailable for PC clusters, Cray XT and K
spec for local view programming (also defined in C).
computer. Source-to- Source compiler to code
int array[YMAX][XMAX];
Code example
with the runtime on top of MPI and GasNet.
#pragma xmp nodes p(4)#pragma xmp template t(YMAX)
data distribution
 HPCC class 2 Winner 2013. 2014#pragma xmp distribute t(block) on p#pragma xmp align array[i][*] to t(i)The spec of XcalableMP 1.x is now converged.
We are now moving to XcalableMP 2.0 with global
task-based parallel programming and PGAS

main(){int i, j, res;
res = 0;add to the serial code : incremental parallelization#pragma xmp loop on t(i) reduction(+:res)for(i = 0; i < 10; i++)for(j = 0; j < 10; j++){work sharing and data synchronizationarray[i][j] = func(i, j);
res += array[i][j];}}3

Example of a Global-view XMP Program Collaboration in Scale project (with Tomita’s ClimateScience Team) Typical Stencil Code!$xmp nodes p(npx,npy,npz)!$xmp template (lx,ly,lz) :: t!$xmp distribute (block,block,block) onto p :: t!$xmp align (ix,iy,iz) with t(ix,iy,iz) ::!$xmp&
sr, se, sm, sp, sn, sl, ...
!$xmp shadow (1,1,1) ::!$xmp&
sr, se, sm, sp, sn, sl, ...
...
!$xmp reflect (sr, sm, sp, se, sn, sl)!$xmp loop (ix,iy,iz) on t(ix,iy,iz)
do iz = 1, lz-1
do iy = 1, ly
do ix = 1, lx
wu0 = sm(ix,iy,iz ) / sr(ix,iy,iz )wu1 = sm(ix,iy,iz+1) / sr(ix,iy,iz+1)
wv0 = sn(ix,iy,iz ) / sr(ix,iy,iz )...
4declare a node arraydeclare and distributea templatealign arrays
add shadow area

stencil communication
parallelize loops

Local-view XMP program: Coarray XMP includes the coarray feature imported from Fortran 2008 for the local-view programming.

 Basic idea: data declared as coarray can be accessed by remote nodes.
 Coarray in XMP/Fortran is fully compatible with Fortran 2008.

b is declared as a coarray.

real b(8)[*]if (xmp_node_num() == 1) thena(:) = b(:)[2]Node 1 gets b from node 2.

 Coarrays can be used in XMP/C.
 The subarray notation is also available as an extension.

5

 Declaration

float b[8]:[*];

 Put

a[0:3]:[1] = b[3:3];

puts b to node 1.

 Get

a[0:3] = b[3:3]:[2];gets b from node 2.

 Synchronization

void xmp_sync_all(int *status)

Preliminary Performance of XcalableMP on Fugaku
 XcalableMP was taken as a parallel programming language project for improving theproductivity and performance of parallel programing.
3000

 Better result than MPI for stencil apps

2500

Speedup (MPI on 8 = 8)

512
256

XMP128

MPI

64

Performance (GFlops) XcalableMP is now available on Fugaku

2000

FugakuOakbridge-CX

1

4#procs

1500
1000
500
03216168832128

512# XMP nodes

Speedup of Impact-3D (global view, stencil apps)
Fusion simulation code
Size = 512^3

Performance Himeno Benchmark(12threads/node, 4 proc/node)Comparing with Cascade Lake
(2.7GHz, 28 core/socket, 2 sockets/node,28 threads/process)6Performance of XcalableMP coarray on Fugaku
 The performance is enhanced by the Fugaku interconnect, Tofu-D, as well as nodeperformance.
 The runtime is implemented using uTofu Libs. (better than MPI)
Note that the reason of the performance degradation of the XMP version on the K computer is theoverhead of allocation for allocatable coarray used as a buffer for communication. It is improved byremoving this overhead by using the uTofu.

QCD (Local view programming, Coarray)

NT-Chem (local view programming, Coarray)
512

512

XMP/K
MPI/K
XMP/Fugaku
MPI/Fugaku

128
64

MPI/K

128

32
16
8
4

XMP/Fugaku

64

MPI/Fugaku

32
168421

XMP/K

256

Speedup (MPI/K on 2 = 2)

Speedup (XMP/K on 1 = 1)

256

1416

# XMP nodes

64

256

2248

# XMP nodes

1632647

新しい言語は難しい？
Pythonとのmixed programming: DSL的に使う

XMP API

並列のカーネルを記述し、それをPythonから呼び出す
言語拡張・コンパイラアプローチでは限界があるの で、XMPとなるべく「同等」なプログラミングができる、ライブラリAPIを定義することを提案したい
XMPの基本的なオブジェクトである、ノード集合、テンプレート、分散配列等を、ディスクリプタで表現し、それに対するdirectiveの操作をAPIで表現する。
プロトタイプ実装は終了、現在公開を予定 （リポジトリにはある？）

Pythonによる並列プログラミング「Scripting XcalableMP = SXMP 分散メモリ並列計算機用スクリプト言語 SXMP の実装と評価」
新井正樹，津金佳祐，前田宗則，吉川隆英（富士通）
情報処理学会 HPC研究会 http://www.ipsj.or.jp/kenkyukai/event/hpc183.html
XMP APIではないが、XMPのruntimeを利用。

Mixed-language programming with Omni XMP compiler

user code (a.c)(a.out)$ xmpcc a.c -o a.out
Add linkage API functions

A user code with XMP directives is translated to a parallel codewith runtime calls of Omni compiler's runtime library
The runtime library is implemented in C and MPI
The translated parallel code is compiled by a native compiler
e.g. GNU, Intel, PGI, Cray, and so on
Linkage API functions are defined for mixed-language
programming
9Serial Python program calls a parallel XMP program
Python program (a.py)

XMP program (xmp.c)

import xmp
lib = xmp.Lib("xmp.so")args = ([1,2,3], [3,4,5])
job = lib.spawn(4, "call_xmp", args, async=True)// other work
job.wait()

void call_xmp(long a1[3],long a2[3]){#pragma xmp nodes p[3]:PythonXMP

xmp.Lib.spawn() method creates new processes and they work as
an XMP program in parallel$ xmpcc -fPIC -shared xmp.c -o xmp.so$ mpirun -np 1 python a.py10

Example: Graph Order/degree solver in Python + XMP

Implementation

The existing python program generates an initial solution and saves a final solution
XMP program searches an optimal solution (includes calculates diameter and ASPL)
Optimization algorithm in XMP uses Simulated Annealing

Python

XMP/C

import xmp:
lib = xmp.Lib("xmp.so")args = (vertices, degrees, edge)
lib.call(MPI.COMM_WORLD, "xmp_sa", args)
void xmp_sa(int vertices, , int degrees,
int edge[vertices*degrees/2][2]){:#pragma xmp loop on t[i]for(int i=0;i<vertices;i++){: // Calculate diameter and ASPL}#pragma xmp reduction(max:diameter)#pragma xmp reduction(+:ASPL)
11

これからの計画


XMP API

公開・マニュアル



C++での利用、wrapper template libraryの設計・開発

XcalableMP 2.0

PGAS上の並列分散タスク並列を中心した仕様
 プロトタイプ実装があるが、開発はslow-downしている。
 現在、富岳でのUCXの実装など、基本的なruntimeの整備をしている状況関連プロジェクト
XcalableACC

XcalableMP + OpenACC、 GPUつきのクラスタ向けの並列プログラミング言語

Omni Compiler

C & Fortranのsource-to-source コンパイラの開発インフラ
 OpenMP 4.x のoffload機能の実装
 OpenMP/OpenACCのOpenCL出力

AMD, POCL, FGPA, RISC-Vへの対応