Omni コンパイラの概要

20

Omni XcalableMP
 理研AICSと筑波⼤で開発中のXMP処理系


XMP/C



XMP/Fortran オープンソース
 トランスレータ + ランタイム(MPIベース)
 OpenACC、XcalableACC対応
OpenMP 4 を計画

 Front-end
Fortran 2008︓ 今年度、ほぼ終了



C++︓ LLVM clangベースで開発中

21

What is “Omni Compiler”?
 Omni compiler is a collection of programs and libraries thatallow users to build code transformation compilers.

 Omni Compiler is to translate C and Fortran programs withdirective-based extensions such as XcalableMP and OpenACC
directives into codes compiled by a native back-end compiler
linked with runtime libraries.

 Supported directives: OpenMP (C, F95)
 OpenACC (C) XcalableMP (C, F95)
 XcalableACC (C)June 8, 201622

Overview of Omni Compiler The source programs in C and Fortran are translated into an XML

intermediate form call XcodeML by the front-end programs, C-front
and F-front.

 Currently, only C and Fortran90 are supported.

 The XcodeML is an intermediate form designed to be translatedback to the source program.

 Exc tools are a collection of Java classes to analyze and transform

programs.

Front-end

C or FortranC or FortranSource program
Source program

De-compiler

XcodeMLIntermediate
File (XML)Native compiler (back-end)

Objectfile
June 8, 2016

Analysis &Transformation(Exc tools)
23

Brief History of Omni Compiler Project 1999-2003: Omni OpenMP compiler project started at RWC (Real World

Computing) project (1992-2002)



OpenMP 1.0 (C and Fortran77 (f2c) ) supported

Cluster-enabled OpenMP (on top of DSM for Distributed memory)Proprietary intermediate code (called Xcode) for C

 (2003:

supported by U. Tsukuba) 2007: XcalableMP Specification Group launched 2008-2011: e-science projectC-front Revised (“gcc” based parser, gnu-extension supported)

F95 parser



XcodeML for C and F95

XcalableMP Spec v 1.0 released

Omni XcalableMP compiler for C released

 2012 - : supported by RIKEN AICS,

2014- : adopted for post-K projectOmni XcalableMP compiler for F95 released

Omni OpenACC (by U. Tsukuba)

2016: Omni XcalableMP compiler v 1.0

June 8, 201624

Example: Omni OpenACC Compiler

sample.c
C with OpenACC
directives

a.out

Omni
Frontend

sample.xml

OpenACC
translator

XcodeML

Omni compiler
infrastructure

acc runtime

sample_tmp.oCcompilersample_tmp.cC with ACC API

sample.gpu.o

nvcc

sample.cu
CUDA

June 8, 201625

Objective and Design of XcodeML
 XML for an intermediate code
 Human-readable format in XML
 It contains all information to translate the intermediate code backto the source program Support for C and Fortran 90 (currently, working also for C++) Syntax errors are supposed to be detected in front-end.
 Separate front-end and transformation phase, and back-end
Rich tools to handle XML can be used.

 Structures
 Type definitions
 Symbol tables for global declarations
 Program (function definitions) in ASTs

June 8, 201626

Exc tool kits: A Java Class Libraryfor Analysis and Transformation
 Currently, the tools consist of the following two Java package.
 Package exc.object: this package defines the data structure.

“Xobject”, which is an AST (Abstract Syntax Tree) for XcodeMLintermediate form. It contains XcodeML input/output, classes andmethods to build and access AST represented by “Xobjects”. It isuseful to scan and analyze the program.

 Package exc.block: this package defines data structure called“Block” to represents block structures of programs. It is useful toanalyze and transform the programs.

 Classes to implement languages
 Pacakge exc.openmp: A translator for OpenMP compiler.
 Package. exc.xcalablemp, exc.xmpf: A translater for XcalableMPcompiler.

June 8, 201627

XcodeMLのデモ

28

Design patterns in exc tool kits (1) XobjectIterator is an iterator to traverse all Xobjects

represented in Xobject class.

 Subclass: bottomupXobjectIterator
 topdownXobjectIterator

XobjectIterator i = new topdownXobjectIterator(x);for(i.init(); !i.end(); i.next(){
x = i.getXobject();...; // some operations …
}June 8, 2016

29

Design patterns in exc tool kits (2) Class XobjectDef is a container of external definitions in

Xobject File.

 Use visitor pattern to access XobjectDef in XobjectFile.
 iterateDef method of XobjectFile traverses XobjectDef.

class MyFuncVistor implements XobjectDefVisitor {public void doDef(XobjectDef d){… do something on definition of d ...
}...
}/* in main routine */XobjectVisitor op = new MyFuncVisitor;
f.iterateFuncDef(op);June 8, 201630

Simple Example using exc tools: print all variable reference Use XobjectIterator class to scan AST
 Use visitor pattern to access XobjectDef in XobjectFile.

class PrintVarRef implements XobjectDefVisitor {public void doDef(XobjectDef d){String fname = d.getName();XobjectIterator i = new topdownXobjectIterator(d.getFunBody());
for(i.init();!i.end(); i.next()){
Xobject x = i.getXobject();if(x.isVariable() || x.isVarAddr())
System.out.println("Variable '"+v.getName()+"' is referenced from Function '"+fname+"'");}}
public static void main(String args[]){
XobjectFile f = new XobjectFile();f.Input("foo.x");
XobjectVisitor op = new PrintVarName();f.iterateFuncDef(op);}June 8, 201631

Block and BlockList in exc.block Data structure for code transformation
 Double-linked list to insert/delete.
 Create new function definition

June 8, 201632

Example 1: Translation of OpenACC parallel construct

June 8, 201633

Example 2: Translation of OpenACC loop construct

June 8, 201634

Projects using Omni Compiler/XcodeML
 XcalableMP XcalableACC (XcalableMP + OpenACC)
 CLAW

June 8, 2016

Fortran Compiler (Meteo Swiss)

35

Current Status and Plan On going: Documentation !!!!!
 Fortran 2003 support

We got support from the vender partner. They provide us test-suite ofF2003.

 C++ support
Using Clang/LLVM front-end to convert Clang AST to XcodeML

 Plan

XMP 2.0 (tasklet construct for multithreaded execution in manycore)

XcalableMP for C++ (object-oriented features)



OpenACC (XcalableACC) for Fortran 2003

June 8, 201636

“メタプログラミング” in Fortran?
 SWoPP 2017で発表

37

June 8, 2016

38

XMPと他の⾔語との連携〜Pythonと連携するPGAS⾔語
XcalableMPのプログラミングモデル〜
Masahiro Nakao, Hitoshi Murai, Taisuke Boku and Mitsuhisa Sato, “Linkageof XcalableMP and Python languages for high productivity on HPC cluster
system ‐ Application to Graph Order/degree Problem”, PGAS‐EI workshop,
HPCAisa 2018.

39

研究目的
PGAS言語の目的は、プログラマの生産性を上げることそのためには、他の言語・ライブラリ・ツールなどとの連携が重要
過去の研究で、BLASやプロファイリングツールとの連携は行った

PythonとXMPの連携機能の作成
Pythonを利用するメリット
既存のPythonライブラリの有効活用
SciPyやNumPyといった科学技術計算用ライブラリが充実
HPCでユーザが多い
40

Omni compilerの構成

Source-to-Source コンパイラ
ユーザが書いた指示文と拡張構文があるコードは、Omni Compilerが用意しているランタイムの呼び出しを含むコードに変換される
Omni Compilerのランタイムは，基本的にはC + MPIを利用

41

MPIプログラムからXMPプログラムを利用する機能
下記関数を実装（xmp.hで定義）

MPIプログラム（mpi.c）#include <xmp.h>#include <mpi.h>
int main(int argc, char **argv){
MPI_Init(&argc, &argv);
xmp_init(MPI_COMM_WORLD);
call_xmp();xmp_finalize();

XMPプログラム（xmp.c）
void call_xmp(){#pragma xmp nodes p[3]:$ xmpcc xmp.c -c$ mpicc mpi.c -c$ xmpcc xmp.o mpi.o -o a.out$ mpirun -np 3 a.out
42

XMPプログラムからMPIプログラムを利用する機能
下記関数を実装（xmp.hで定義）

XMPプログラム（xmp.c）#include <xmp.h>#include <mpi.h>#pragma xmp nodes p[3]int main(int argc, char **argv){
xmp_init_mpi(&argc, &argv);
MPI_Comm comm = xmp_get_mpi_comm();
call_mpi(comm);
xmp_finalize_mpi();

MPIプログラム（mpi.c）#include <mpi.h>void call_mpi(MPI_Comm comm){
int rank, size;
MPI_Comm_rank(comm, &rank);
MPI_Comm_size(comm, &size);$ xmpcc xmp.c -c$ mpicc mpi.c -c$ xmpcc xmp.o mpi.o -o a.out$ mpirun -np 3 a.out
43

XMPとPythonの連携機能
1. 並列Pythonプログラム（mpi4py）から、並列XMPプログラムを呼び出す

前ページまでのMPIとの連携と概念は同じ（SPMD）
PythonXMP

2. 逐次Pythonプログラムから、並列XMPプログラムを呼び出す
1つの逐次プログラムが並列プログラムを呼び出す

44

並列Pythonプログラムから、並列XMPプログラムを呼び出す
PythonのXMPパッケージを作成．下記の関数を提供
Function

Description

init(sharedFile, MPI.Intracomm)

XMP環境の初期化

finalize()

XMP環境の完了処理

Pythonプログラム（a.py）
import xmp
from mpi4py import MPI
import numpy # 引数が必要な場合だけ
lib = xmp.init('xmp.so', MPI.COMM_WORLD)
arg = numpy.array([1,2])
lib.call_xmp(arg.ctypes)
xmp.finalize()

XMPプログラム（xmp.c）
void call_xmp(long arg[2]){#pragma xmp nodes p[3]:$ xmpcc -fPIC -shared xmp.c -o xmp.so$ mpirun -np 3 python a.py
45

逐次Pythonプログラムから、並列XMPプログラムを呼び出す
PythonのXMPパッケージを作成．下記の関数を提供
Function

Description

spawn(sharedfile, nodes, funcName, arguments)

XMPプログラム実行

Pythonプログラム（a.py）

XMPプログラム（xmp.c）
void call_xmp(long arg[2]){#pragma xmp nodes p[3]:import xmp
import numpy # 引数の必要な場合だけ

3プロセスで動作する

arg = numpy.array([1,2])
xmp.spawn("xmp.so", 3, "call_xmp", arg)

1プロセスで動作する$ xmpcc -fPIC -shared xmp.c -o xmp.so$ mpirun -np 1 python a.py
1プロセスで実行
46

Graph Order/degree問題とは無向グラフにおいて与えられた頂点数と次数を持つ「直径」と「頂点間の平均距離」を最小化する問題のこと（低遅延な相互結合網の設計に役立つ [藤原 2015]）
頂点数と次数から，最小の直径と平均距離は算出できる[V. G. Cerf 1974]が，どのようなグラフになるかはわからない（おそらくNP困難）
頂点数=10，次数=3の場合の例：
3
2

2

1
1

直径=3，平均距離=1.89（ランダム）

直径=2，平均距離=1.67（最適解）47

Graph Golf@NII
Graph Order/degree問題に対するコンペティションをNIIが開催
2015年から毎年開催（http://research.nii.ac.jp/graphgolf）
複数の頂点数と次数の組合せを出題し，最小の直径と平均距離を出したグループを表彰する

頂点数=32，次数=5頂点数=256，次数=1848

XMPとPythonを組み合わせる利点
Graph Golfの公式サイトで，本問題に対するPythonプログラムが公開されている
http://research.nii.ac.jp/graphgolf/py/create-random.py （100行くらい）
下記を出力する
ランダムな初期解
ある解の直径と平均距離の出力ある解をPNG形式で保存
Pythonのnetworkxパッケージが利用されている

直径=3，平均距離=1.89（ランダム）課題
グラフの初期解を生成するが，最適解の探索は行わない
直径と平均距離の算出に時間がかかる
この2つの課題に対する機能をXMPを使って作成する
49

Graph Order/degree solver in Python + XMP
実装方針
初期解の生成と図の出力は，既存のPythonコードを利用
最適解の探索（直径と平均距離の計算を含む）はXMPを利用
最適解探索はSimulated Annealingを用いる
Python import xmp:
xmp.init("xmp.so", MPI.COMM_WORLD)
lib.sa(c_int(lines), edge, c_int(vertices))
xmp.finalize()

XMP/C void sa(int lines, int edge[lines][2], intvertices){:#pragma xmp loop on t[i]for(int i=0;i<vertices;i++){: // 各頂点における直径と平均距離を: // 幅優先探索で算出}#pragma xmp reduction(max:diameter)#pragma xmp reduction(+:average_distance)
}

50

評価環境
COMA@筑波大を利用

1回の直径と頂点間平均距離を計算するのに要する時間を測定
問題サイズは頂点数=9344，次数=1051

結果
flat-MPIで実行
20プロセスまで1ノード内
オリジナルのPythonコード（networkxパッケージ）で要する時間は148.83秒

921倍の
高速化
Number of Processes
52

XcalableMP 2.0 に向けて
〜 分散メモリ上タスク並列モデル 〜••「PGAS モデルに基づく⼤規模並列クラスタ向け⾼⽣産並列プログラミングモデルに関する研究」筑波⼤学 津⾦佳祐 学位論⽂
Keisuke Tsugane, Jinpil Lee, Hitoshi Murai, and Mitsuhisa Sato. “Multi‐tasking Execution in PGAS L
anguage XcalableMP and Communication Optimization on Many‐core Clusters. In HPC Asia 2018:
International Conference on High Performance Computing in Asia‐Pacic Region, January 28–31, 2
018, Chiyoda, Tokyo, Japan.
53

タスク並列プログラミングモデル
 共有メモリ向けの様々なタスク並列モデル
From PLASMA/QUARK slides
by ICL, U. Teneessee

Cilk Plus，TBB，OpenMP

 データ依存に基づくタスク並列モデルの登場

OpenMP 4.0，Quark，StarPU，
OmpSs

 コアの多いメニーコアプロセッサにて⾼い性能が期待される



全体同期からデータ依存によるタスク間の細粒度な同期へ
通信と演算のオーバラップが容易

Execution results of Cholesky
Factorization in OpenMP 4.0 task

54

OpenMP task 指⽰⽂
 仕様 4.0 からはタスク間の依存関係を記述可能


depend 節 の in，out，inout にデータ依存を記述

逐次実⾏を基にした依存モデル（フロー，出⼒，反依存）異なるノードの並列動作するスレッド上のタスク依存⽣成は困難

taskA

taskB
flow dep.

output dep.

taskC
anti dep.

taskD#pragma omp parallel#pragma omp single{
int A, B, C;#pragma omp task depend(out:A)
A = 1;/* taskA */#pragma omp task depend(out:B)
B = 2;/* taskB */#pragma omp task depend(in:A, B) depend(out:C)
C = A + B;/* taskC */#pragma omp task depend(out:A)
A = 2;/* taskD */}55

ノード間のタスク依存
 通信による依存関係の指定


ノード間の依存関係 = データの移動（通信）と定義



通信をタスク化し通信対象をデータ依存とする


依存関係が満たされれば（通信が終了すれば）演算タスクが実⾏

out(A[0])in(A[0])out(A[1])int A[2];#pragma omp parallel#pragma omp single{#pragma omp task depend(out:A[0])funcA(A[0]);#pragma omp task depend(in:A[0]) depend(out:A[1])funcB(A[0], A[1]);}56ノード間のタスク依存 (contʼd) 通信による依存関係の指定


ノード間の依存関係 = データの移動（通信）と定義



通信をタスク化し通信対象をデータ依存とする


依存関係が満たされれば（通信が終了すれば）演算タスクが実⾏

rank 0rank 1out(A[0])MPI_Recv(B)in(A[0])Comm.

MPI_Send(A[0])out(B)in(B)
out(A[1])int A[2], B;
if (rank == 0) {#pragma omp task depend(out:A[0])funcA(A[0]);#pragma omp task depend(in:A[0])MPI_Send(A[0], 1, …);Send は in

Recv は out} else if (rank == 1) {#pragma omp task depend(out:B)
MPI_Recv(B, 0, …);#pragma omp task depend(in:B) depend(out:A[1])funcB(B, A[1]);}57

XMP tasklet 指⽰⽂の記述案#pragma xmp tasklet [clause[, ...]] [on {node-ref | template-ref}](structured-block)where clause is :{in | out | inout} (event_variable[, …])
 実⾏モデル

OpenMP のタスク依存と同様



各ノードがそれぞれタスクを⽣成，並列実⾏

tasklets 指⽰⽂で囲われた範囲のみ並列実⾏

out(A)
flow dep.
outputdep.

on 節による実⾏ノードの指定 ノード内/間のタスク依存の記述が必要

ノード内は OpenMP

ノード間は通信で記述


XMP の各モデルの通信を依存関係として使えるように拡張

in(A)anti dep.

out(A)

58

グローバルビューにおけるtasklet 指⽰⽂#pragma xmp tasklet gmove [clause[, ...]] [on {node-ref | template-ref}](an assignment statement)
where clause is :{in | out | inout} (event_variable[, …])
 tasklet gmove，tasklet reflect 指⽰⽂を提案
 タスク内で gmove を実⾏


配列代⼊⽂に分散配列が含まれる場合に通信が発⽣



通信相⼿は実⾏ノード集合内で決定

実⾏ノード集合内での暗黙の同期を含まない

基本的に 1 対 1 通信とし，同期は各通信単位で実⾏

 tasklet 指⽰⽂同様に in，out，inout 節を提供

59

グローバルビューにおけるtasklet 指⽰⽂の例
int A[2], B;#pragma xmp nodes P(2)#pragma xmp template T(0:1)#pragma xmp distribute T(block) onto P#pragma xmp align A[i] with T(i)Node 1

Node 2out(A[0])in(A[0])out(A[1])#pragma xmp tasklets{
ノード1 の A[0] から
ノード2 の A[1] へ通信#pragma xmp tasklet out(A[0]) on P(1)A[0] = 0;/* taskA */#pragma xmp tasklet gmove in(A[0]) out(A[1]) on P(1:2)
dependency
A[1] = A[0]; /* taskB */#pragma xmp tasklet in(A[1]) out(B) on P(2)communication
B = A[1];/* taskC */}in(A[0])out(A[1])in(A[1])out(B)

taskA
taskB
taskC

60

ローカルビューにおけるtasklet 指⽰⽂#pragma xmp tasklet [clause[, ...]] [on {node-ref | template-ref}](an assignment statement)
where clause is :{put | get} (tag)or{put_ready | get_ready} (variable, {node-ref | template-ref}, tag) タスク内でユーザ記述の coarray を実⾏可能


タスク間の⽚側通信なので，”通信可能通知”，”⽚側通信の実⾏（coarray）”，”通信完了通知”が必要 ⽚側通信に合わせた，”通信可能通知”/”通信完了通知”を記述する節を提供（put_ready/put，get_ready/get 節）61ローカルビューにおけるtasklet 指⽰⽂の例
 ノード2 の taskB が ノード1 の変数 A に Put を実行#pragma xmp nodes P(2)int A:[*], B, C, tag;
#pragma xmp tasklets{#pragma xmp tasklet in(A) out(B) on P(1)B = A;/* taskA */#pragma xmp tasklet out(A) put(tag) on P(2)A:[1] = 1; /* taskB */#pragma xmp tasklet in(A) out(D) ¥put_ready(A, P(1), tag) on P(1)C = A;/* taskC */}Node 1in(A)put_ready Nodeの実⾏開始通知を2
早く出すことで Put 通信の開始を早めることが可能

out(A)out(A)
taskA

Put
in(A)

taskB

dependency

taskC

communication
62

試作実装
 Omni XMP コンパイラに tasklet 指⽰⽂を実装


MPI+OpenMP へのコード変換
軽量スレッドライブラリ Argobots＠ANL による実装︓任意地点で実⾏可能なタ
スクスイッチの機能を実装


スレッド内における通信と計算のオーバラップによる⾼速化

 MPI+OpenMP コードへの変換


提案指⽰⽂を対応する OpenMP 指⽰⽂へと変換

tasklet 指⽰⽂->

task 指⽰⽂

基本的に XMP ランタイムの gmove，coarray 実装を使⽤

MPI 通信は以下のように変更

MPI_Send(…);
or
MPI_Isend(…);
MPI_Wait(…);

MPI_Isend(…);
MPI_Test(&comp, …);
while (!comp) {#pragma omp taskyield
MPI_Test(&comp, …);}

⾮同期に通信完
了を確認

別のタスクへスイッチ 63評価︓実験環境
 実験環境

Oakforest-PACS（最先端共同 HPC 基盤施設 : JCAHPC）

 評価項⽬

ブロックコレスキーベンチマーク



問題サイズ : 32k × 32k，ブロックサイズ : 512 × 512



最⼤ 32 ノード，64 スレッド / ノード で実⾏
CPUIntel Xeon Phi 7250
（68 cores）

Memory16 GB（MCDRAM）
96 GB（DDR4）

Interconnect

Intel Omni-Path Architecture

SoftwareIntel Compiler ver. 17.0.1
Intel MPI library 2017 update 1
64

Example Block Cholesky

Factorization

double A[nt][nt][ts∗ts], B[ts∗ts], C[nt][ts∗ts];#pragma xmp nodes P(∗)
#pragma xmp template T(0:nt−1)#pragma xmp distribute T(cyclic) onto P#pragma xmp align A[∗][i][∗] with T(i)for (int k = 0; k < nt; k++) {#pragma xmp tasklet out(A[k][k]) ¥get ready(A[k][k], T(k:), k∗nt+k) on T(k)potrf(A[k][k]);#pragma xmp tasklet in(A[k][k]) out(B) get(k∗nt+k) on T(k:)#pragma xmp gmove in
B[:] = A[k][k][:];for (int i = k + 1; i < nt; i++) {#pragma xmp tasklet in(B) out(A[k][i]) ¥get ready(A[k][i], T(i:), k∗nt+i) on T(i)trsm(A[k][k], A[k][i]);}for (int i = k + 1; i < nt; i++) {#pragma xmp tasklet in(A[k][i]) out(C[i]) get(k∗nt+i) on T(i:)#pragma xmp gmove inC[i][:] = A[k][i][:];for (int j = k + 1; j < i; j++) {#pragma xmp tasklet in(A[k][i], C[j]) out(A[j][i]) on T(j)gemm(A[k][i], C[j], A[j][i]);}#pragma xmp tasklet in(A[k][i]) out(A[i][i]) on T(i)syrk(A[k][i], A[i][i]);}}#pragma xmp taskletwait#pragma xmp barrier
65ブロックコレスキーベンチマークの結果
ループ並列とタスク並列の⽐較



MPI+OMP (Parallel Loop) と MPI+OMP (task)

XMP と MPI+OMP のタスク版の⽐較



XMP global-view/local-view と MPI+OMP (task)

Performance [GFLOPS]

better

10000
8000
6000

MPI+OMP, Parallel Loop
MPI+OMP, Task
XMP globalview (gmove)
XMP localview (put)

4000
2000
0
1

2

4

8

16

32

# of nodes
66メニーコア（OFP)における最適化
 MPI_THREAD_MULTIPLEが遅かったので、通信を１スレッドに集めた（opt) Argobotで、スレッドのスイッチをできるようにした。

Performance [GFLOPS]

OpenMPのthread_yieldは効かない。

9000
8000
7000
6000
5000
4000
3000
2000
1000
0

MPI+OMP, Task
MPI+OMP, Task (opt)
MPI+ABT, Task
MPI+ABT, Task (opt)

1

2

4

8

16

32

# of nodes
67これからのPGASプログラミング
モデルの動向PGAS applications workshop (PAW) at SC17, Keynote “How does PGAS “collaborate” with MPI+X?” by M. Sato, より。

68

“MPI+X” for exascale?
 X is OpenMP!
 “MPI+Open” is now a standard programming for high-end systems.

Iʼd like to celebrate that OpenMP became “standard” in
HPC programming Questions:“MPI+OpenMP” is still a main programming model for
exa-scale?

 How does PGAS “collaborate” with MPI+X?


Can PGAS replace(beat!?) “MPI” for exascale?

69誤解︓すべてをPGASで書かなくてはならないわけではない
 MPI broadcast operation can be

written in CAF easily, …

 But, is it efficient?

 Should use “co_broadcast” in

CAF.



Sophisticated collective
communication libraries of“matured” MPI are required Obviously, PGAS need tocollaborate with MPI.

REAL(8),DIMENSION(:),ALLOCATABLE :: ARRAY・・・
CALL MPI_COMM_SIZE(MPI_COMM_WORLD,NIMG,IERR)CALL MPI_COMM_RANK(MPI_COMM_WORLD,ID,IERR)
・・・
CALL MPI_BCAST(ARRAY, MSGSIZE, MPI_REAL8, 0,
MPI_COMM_WORLD,IERR)

REAL(8),DIMENSION(:),CODIMENSION[:],ALLOCATABLE :: ARRAY・・・
NIMG= NUM_IMAGES()
ID= THIS_IMAGE()
・・・
SYNC ALLIF(ID /= 1) THENARRAY(1:MSGSIZE) = ARRAY(1:MSGSIZE)[1]END IFSYNC ALL70

PGAS とremote memory access (RMA)/one-sided comm.の違い


PGAS is a programming model relating to distributed memory system witha shared address space that distinguishes between local (cheap) andremote (expensive) memory access.
Easy and intuitive to describe remote data access, for not only one side-comm,but also stride comm.

RMA is a mechanism (operation) to access data in remote memory bygiving address in (shared) address space.


RDMA is a mechanism to directly access data in remote memory without involving
the CPU or OS at the destination node.
Recent networks such as Cray and Fujitsu Tofu support remote DMA operation
which strongly support efficient one-sided communication.

PGAS is implemented by RMA providing light-weight one-sided
communication and low overhead synchronization semantics.



For programmers, both PGAS and RMA are programming interfaces andoffer several constructs such as remote read/write and synchronizations.


MPI3 provides several RMA (one-sided comm.) APIs as library interface.

71

PGASは、MPIに勝てるのか︖
Can PGAS replace MPI? / Can PGAS be faster than MPI?
 Advantages of RMA/RDMA Operations







(Note: Assume MPI RMA is an API for PGAS)
multiple data transfers can be performed with a single
synchronization operation
Some irregular communication patterns can be more
economically expressed
Significantly faster than send/receive on systems withhardware support for remote memory access

Recently, many kinds of high-speed interconnect have
hardware support for RDMA, including Infiniband, … as well as
Cray and Fujitsu.

72

Case study: “Ping-pong” in CAF and RDMA
Ping-pong: single data transfer
with synchronization operation
 EPCC Fortran Coarray microbenchmark
image1

image2Ping
phasex(1:ndata)[image2] = x(1:ndata)sync images

sync images

Pong
phase

sync images

x(1:ndata)[image1] = x(1:ndata)sync images

Bandwidth of PUT-based achieves almost the same performanceof send/recv. But, latency of small messages is bad.



PUT exec time = data transfer + wait ack + sync_image


Sync_images exchange sync between nodes. It can be optimized by
one-direction sync: post/wait (XMP), notify/query (Rice U, CAF2.0),
event post/event wait (Fortran 2015)
PRIMEHPC FX100 (FJ-RDMA)

bandwidth

latency73

Fiber min-apps in CAF/XMPHitoshi Murai, Masahiro Nakao, Hidetoshi Iwashita and Mitsuhisa Sato,"Preliminary Performance Evaluation of Coarray-based Implementation ofFiber Miniapp Suite using XcalableMP PGAS Language“, PAW2017 (inmorning)Replace of MPI with Coarray operation by simple
rewriting rule.



Our coarray-based implementations of three (NICAMDC, NTChem-MINI, and FFB-MINI) of the five miniapps
were comparable to their original MPI implementations.



However, for the remaining two miniapps (CCS QCD
and MODYLAS-MINI) due to communication buffer
management

Simple re‐writing rule

Result of
NICAM‐DC74

Case study(2): stencil communication
Typical communication pattern in domaindecomposition.

Advantage of PGAS: Multiple data transfers with asingle synchronization operation at end



PUT non-blocking outperforms MPI in Himeno
Benchmark!
Donʼt wait ack before sending the next data (by FJ-RDMA)

NOTE: The detail of this results is
to be presented in HPCAisa 2018:
Hidetoshi Iwashita, Masahiro
Nakao, Hitoshi Murai, Mitsuhisa
Sato, “A Source‐to‐Source
Translation of Coarray Fortranwith MPI for High Performance”
75

PGASの得意な通信パターン
When the communication pattern isexpressed in some defined forms, PGAS
comm. can be optimized.


Coarray assignment by array section
syntax. Runtime select better stride
comm, pack/unack or direct RDMA.

A(1, 1:10)[1] = B(1:10)
shadow: declares width of shadow reflect: executes stencil comm.
 width: specifies shadow width to be
updated. /periodic/: updates shadow
periodically. async: asynchronous comm.
 wait_async: completes async. reflects// put from image2 to image1

Reflect operation in XMP global viewmodel (stencil neighbor comm)

Optimized stencil
communication by reflect
directive on K computer

Optimized by RDMA
with pt‐to‐pt sync.

600



Irregular Communication Patterns:

If communication pattern is not known a priori,
but the data locations are known, send-recv
program needs an extra step to make pairs ofsend-recv. BUT, RMA can handle it easily
because only the source or destination processneeds to perform the put or get call

Speedup (single=1)

500

400

Pact/unpack withbarrier sync.

300

200

RDMA with barrier
sync.

100

0
0

200

400

600

800

1000

1200

Number of Compute Nodes

76
Target: a prototype dynamical core of a climate model SCALE‐LESMPI RMA は、PGASの低レベルの通信レイヤとして適当か︖
 “MPI is too low and too high API for communication”. (Prof.

Marc Snir, JLESC 7th WS)MPI RMA APIs offer their PGAS model rather than “primitives” forother PGAS.

 In case of our XMP Coarray implementation:Using “passive target”
MPI flush operation and synchronization do not match to
implement “sync_images”.
Complex “window” management to expose the memory as acoarray.

(We need more study for better usage of MPI RMA)

Fujitsu RDMA interface is much faster in K-computer.

77

“Compiler-free” アプローチ︓最近のトレンド Library approach: MPI3 RMA, OpenShmem, GlobalArray, …
 C++ Template approach: UPC++, DASH, …
 This approach may increase portability, clean separation from basecompiler optimization, … but sometimes hard to debug in C++
template…

 But, approach by compiler will give:New language, or language extension provides easy-to-use andintuitive feature resulting in better productivity.
Enable compiler analysis for further optimization: removal ofredundant sync and selection of efficient communication, etc, …
But, in reality, compiler-approach is not easy to be accepted for
deployment, and support many sites, …
78