XcalableMP

Directive-Based Language eXtension for
Scalable Parallel Programming
What’s XcalableMP?
MPI is widely used as a parallel programming model.
However, the programming cost of MPI is high.
XcalableMP[1-3], XMP for short, is a directive-based
language extension which allows users to develop
parallel programs for distributed memory systems
easily and to tune its performance by having minimaland simple notations.
XMP specification is being designed by XcalableMPSpecification Working Group(XMP-WG). XMP-WG isa special interest group, which is organized to makea draft on “petascale” parallel language. XMP-WG
consists of members from academia(U. Tsukuba, U.
Tokyo, Kyoto U. and Kyusyu U.), research labs
(RIKEN, NIFS, JAXA, JAMSTEC/ES) and industries(Fujitsu, NEC, Hitachi) in Japan.
X M P prototype compiler and tools are b e i n gdeveloped in "Seamless and Highly-productive
Parallel Programming Environment for Highperformance computing" project funded by Ministry ofEducation, Culture, Sports, Science and Technology,
JAPAN.

Features
To reduce code writing and educational costs
Language extension of C99 and Fortran 95

Current Solution for parallel programming
int array[MAX];
main(int argc, char **argv){
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);dx = MAX/size;
llimit = rank * dx;
if(rank != (size -1)) ulimit = llimit + dx;
else ulimit = MAX;

Only way to program is MPI,but MPI programming seems
difficult,... we have to rewrite
almost entire program and it
is time-consuming and hard to
debug... mmm

temp_res = 0;
for(i=llimit; i < ulimit; i++){array[i] = func(i);temp_res += array[i];}}MPI_Allreduce(&temp_res, &res, 1, MPI_INT, MPI_SUM, ...);
MPI_Finalize( );We need better solutions !!
int array[MAX];#pragma xmp template t(0:MAX-1)#pragma xmp nodes p(*)#pragma xmp distribute t(block) onto p#pragma xmp align array[i] with t(i)main(){#pragma xmp loop on t(i) reduction (+:res)for(i = 0; i < MAX; i++){array[i] = func(i);res += array[i];}}XcalableMP enables users toeasily develop parallel
programs and to tuneperformance with minimal andsimple notation !!

Status
XMP specification version 1.0 is available
Omni XMP compiler 0.5.3 for C is available fromUniversity of Tsukuba

Supports typical parallelization based on data
parallel paradigm and work mapping under“global-view” programming model

Download from http://www.xcalablemp.org

Also includes Coarray Fortran like feature as“local-view” programming model

For accelerators(GPU, etc)

Many concepts are inherited from High Performance Fortran
Performance awarenessCommunication,synchronization andwork-mapping occur
when directives areencountered

Interface of Scalasca & tlog profiling tools
XMP Parallel I/OInterface of MPI library

work in progress

XMP will be used to program to K computer

Programming Model

Execution model is a Single Program Multiple
Data (SPMD)
A thread starts
execution in eachnode independently(as in MPI)

Supported platforms are Linux cluster, Cray platform, ..

Node 1

Node 2

Node 3

Directives
Comm, sync, and work-mapping

All actions are taken by directives for being“easy-to-understand” in performance tuning(different from High Performance Fortran)

Global-view Programming
The global-view programming model supports typical
work-mapping and communication for parallelprograms.
First, to parallelize a program in the global-viewmodel, data distribution is described with a template.
The template is a dummy array used to express anindex space associated with an array. The conceptof the template is shown in the next page’ s figure.
The template and a node set are defined by atemplate directive and a node directive. Distribution of
the template is described by a distribute directive.
Finally, data distribution of the array is specified byan align directive to align the array with the template.

#pragma xmp template t(0:N-1)#pragma xmp nodes p(4)template t
index 0N-1

#pragma xmp distribute t(block) onto p
node 1index 0node 2

N/4-1

node 3

N/2-1

node 4

3*N/4-1

N-1#pragma xmp align array[i] with t(i)array[]
index 0node 1

node 2

node 3

node 4

node 1

node 2

node 3

node 4

N/4-1

N/2-1

3*N/4-1

N-1

1 int array[N];2
3 #pragma xmp template t(0:N-1)
4 #pragma xmp nodes p(4)5 #pragma xmp distribute t(block) onto p
6 #pragma xmp align array[i] with t(i)78 main(void){
9int i, res = 0;
10
11 #pragma xmp loop on t(i)12 for(i = 0; i < N; i++){13array[i] = func(i);14res += array[i];15}16
17 #pragma xmp reduction (+:res)
18 }Above right figure shows a simple example of theglobal-view programming model in XMP C language.
Line 3 defines the template with a start index (0) anda length of index (N). Line 4 defines the node set (4nodes in this case). Line 5 specifies the data
distribution of the template on the node set as a block
distribution. XMP supports block, cyclic, block-cyclic, andgen-block distributions. Line 6 defines the distribution
to align the array with the template.
In line 11, a loop directive is used to parallelize a loop
statement from Line 12 to Line 15. In this case, node
1 executes iterations from 0 to N/4 – 1, and node 2
executes iterations from N/4 to N/2 – 1,independently. Line 17 executes a reduction
operation on the variable res by a reduction directive.
The reduction directive supports typical operators(such as “+” , “MAX” , and so on).
XMP also supports rich communication andsynchronize directives such as “shadow” , “gmove” ,and “barrier” . Global-view communication directives
are used to maintain the consistency of shadow area,
move distributed data globally, and synchronize
nodes.
Local-view Programming
The local-view programming model attempts toincrease performance by considering inter-nodecommunication and local memory of each node
explicitly. In this model, local data distributed on thenode can be referred to using node number.
XMP adopts coarray notation as an extension of
Fortran and C languages for local-view programming.
In the case of Fortran as the base language, thenotation of XMP coarray is compatible with that ofCoarray Fortran. To use coarray notation in C, we
propose a language extension of the C language. To
access coarray, the node number is specified byfollowing an array reference by “:” .
array_name[start : length[:step]]:[node_number]double a[5], b[5];#pragma xmp coarray a // Declartion:b[0:2] = a[3:2]:[2];// Communication

a[]

b[]

Node 1
Node 2

The array_name[start:length]:[node_number] means
elements from the array_name[start] to thearray_name[start+length-1] located on compute node
whose name is node_number.
XMP recommends users to choice the global-viewprogramming model and the local-view programming
model according to an algorithm.

ExampleLaplace Solver by Global-view programming#pragma xmp template t(0:YSIZE-1, 0:XSIZE-1)#pragma xmp nodes p(N_Y, N_X)
#pragma xmp distribute t(block,block) onto p#pragma xmp align u[y][x] with t(x, y)#pragma xmp align uu[y][x] with t(x, y)#pragma xmp shadow uu[1:1][1:1]:#pragma xmp loop (x, y) on t(x, y) threadsfor(y = 1; y < YSIZE-1; y++)for(x = 1; x < XSIZE-1; x++)uu[y][x] = u[y][x];#pragma xmp reflect uu#pragma xmp loop (x, y) on t(x, y) threadsfor(y = 1; y < YSIZE-1; y++)for(x = 1; x < XSIZE-1; x++)u[y][x] = (uu[y-1][x] + uu[y+1][x] +uu[y][x-1] + uu[y][x+1])/4.0;

Defines two dimensional
process gridDefines shadow area
and its width
Specifies additional
thread parallelization
Synchronizes data onlyon shadow area

The laplace solver is the implementation of a laplace
equation using Jacobi iteration with 4 points-stencil
operations. This code is an example to use shadow
and reflect directives which communicate andsynchronize only overlapped region. Moreover,thread parallelization is used in loop statements.
XMP can program simply by using shadow/reflect
operations with keeping serial code image.
Integer Sort of NPB by Local-view programming
int key[SIZE];#pragma xmp coarray key:#pragma xmp barrierfor( i=0; i<comm_size; i++ )key[recv_displ[i]:count[i]]:[i]= buff[send_displ[i]:count[i]];Defines co-array
Exchanges data byusing coarray

The Integer Sort benchmark tests a sorting operationthat is important in particle method codes. This codeis an example to use coarray function which
exchanges data. XMP coarray function will use
efficient one-sided communication library GASNet,
ARMCI, and so on.

Reference[1] Jinpil Lee, Minh Tuan Tran, Tetsuya Odajima, Taisuke
Boku and Mitsuhisa Sato: An Extension of XcalableMPPGAS Language for Multi-node GPU Clusters, NinthInternational Workshop on Algorithms, Models and Tools
for Parallel Computing on Heterogeneous Platforms(HeteroPar 2011), 2011.
[2] Jinpil Lee and Mitsuhisa Sato: Implementation andPerformance Evaluation of XcalableMP: A Parallel
Programming Language for Distributed Memory Systems,
The 39th international Conference on Parallel Processing
Workshops (ICPPW10), pp.413-420, 2010.
[3] Masahiro Nakao, Jinpil Lee, Taisuke Boku, Mitsuhisa
Sato. “XcalableMP Implementation and Performance ofNAS Parallel Benchmarks” , Fourth Conference on
Partitioned Global Address Space Programming Model(PGAS10), Oct., 2010.

For more information, please visit
T2K Open Supercomputer Alliance (#5007@Level 6)
Center for Computational Sciences, University ofTsukuba (#923@Level 4)
http://www.xcalablemp.org