XcalableMP
Directive-based Language eXtension for Scalable andPerformance-aware Parallel Programming
University of Tsukuba
What's XcalableMP?

Although MPI is a de-facto standard for parallel programming on
distributed memory systems, writing MPI programs is often
time-consuming and complicated process. XcalableMP is adirective-based language extension which allows users todevelop parallel programs for distributed memory systems easily
and tune the performance by having minimal and simple
notations. So far, there have been a number of parallel
programming languages for distributed memory architectures,
including High Performance Fortran (HPF). However, these
programming models have not been commonly used any more.
On the other hand, OpenMP is widely used on shared memoryarchitectures including SMP-configured PC clusters or multi-coreCPUs. The most important feature of OpenMP from theperspective of programmability is to enable parallelization withsimple directives that helps users extend their codes relatively
easily from sequential ones. The target platform of OpenMP is
however limited to shared memory architectures.

Current solution for programming clusters?!
int array[YMAX][XMAX];

Only way to program is MPI,
but MPI programming seems
difficult, … we have torewrite almost entire programand it is time-consuming andhard to debug… mmm

main(int argc, char**argv){
int i,j,res,temp_res, dx,llimit,ulimit,size,rank;

MPI_Init(argc, argv);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);dx = YMAX/size;
llimit = rank * dx;
if(rank != (size - 1)) ulimit = llimit + dx;
else ulimit = YMAX;
temp_res = 0;
for(i = llimit; i < ulimit; i++)for(j = 0; j < 10; j++){array[i][j] = func(i, j);
temp_res += array[i][j];}MPI_Allreduce(&temp_res, &res, 1, MPI_INT, MPI_SUM, MPI_COMM_WORLD);MPI_Finalize();}mapping for loop on parallel processes, which are normally
executed as MPI processes. These features can be coded using
directives similar to OpenMP.
The specification has been being designed by XcalableMPSpecification Working Group which consists of members fromacademia and research labs to industries in Japan.

XcalableMP Application Program Interface (XcalableMP API) is acollection of compiler directives, runtime library routines that canbe used to specify distributed-memory parallel programming in Cand Fortran program. This specification provides a model ofparallel programming for distributed memory multiprocessor
systems, and the directives extend the C and Fortran baselanguages as to describe distributed memory parallel program,as in OpenMP.

Features of XcalableMP are summarized as follows:

XcalableMP introduces simple, but effective features to describe
typical scientific applications using a concept similar to OpenMP.
This paradigm features functions for array distribution and work

• The important design principle of XcalableMP is"performance-awareness". All actions of communication andsynchronization are taken by directives, different fromautomatic parallelizing compilers. The user
should be aware of what happens by
XcalableMP directives in the execution modelon the distributed memory architecture.

www.xcalablemp.org

We need better solutions!!
#pragma xmp template T[10]#pragma xmp distributed T[block]We want better solutions

to enable step-by-stepdata …
distribution

int array[10][10];#pragma xmp aligned array[i][*] to T[i]parallel programming fromthe existing codes, … easy-add to the serial code :main(){
to-useand easy-to-tuneincremental
parallelization
int i, j, res;
res = 0;
performance … portable …
#pragma xmp loop on T[i] reduction(+:res)good for beginners.
for(i = 0; i < 10; i++)work sharing and datafor(j = 0; j < 10; j++){array[i][j] = func(i, j);synchronization
res += array[i][j];}}• XcalableMP supports typical parallelization based on the data
parallel paradigm and work mapping under "global viewprogramming model", and enables parallelizing the original
sequential code using minimal modification with simple
directives, like OpenMP. Many ideas on "global-view"programming are inherited from HPF (High PerformanceFortran).

• XcalableMP also includes CAF-like PGAS(Partitioned Global Address Space) feature as"local view" programming.
• Extention of existing base languages withdirectives is useful for rewriting cost andeducation cost. XcalableMP APIs are definedon C and Fortran 95 as a base language.
• For flexibility and extensibility, the execution
model allows to combine with explicit MPI
coding for more complicated and tuned
parallel codes and libraries. For multi-core andSMP clusters, OpenMP directives can be
combined into XcalableMP for threadprogramming inside each node as a hybrid
programming model.(Under discussion)Programming model of XcalableMP

The parallel execution model of XcalableMP is a Single Program
Multiple Data (SPMD) model. As in MPI, all parallel processes innodes start their execution from the same main function. Whenthe thread encounters XcalableMP directives, thesynchronization and communication occurs between nodes. Thatis, no synchronization and communications happen withoutdirectives. In this case, the program does duplicated execution of
the same program on local memory in each node. As default,
data declared in the program is allocated in each node, and isreferenced locally by threads executed in the node.
The global-view programming model is useful when, startingfrom sequential version of the program; the programmer
parallelizes it in data-parallel model by adding directives
incrementally with minimum modifications. As these directives
can be ignored as a comment by the compilers of baselanguages (C and Fortran), an XcalableMP program derived
from a sequential program can preserve the integrity of original
program when it is run sequentially.
The global-view programming model shares major concepts withHPF. Nodes directive declares a node array to express a set ofnodes. The programmer describes the data distribution of data
shared among the nodes by data distribution directives. To
specify the data distribution, the template is used as a dummyarray distributed on nodes. A distributed array is declared byaligning the array to the template by align directive.
The work mapping in loop iteration is described by the loopdirective as in OpenMP. Loop construct maps iterations to thenode where referenced data is located. Template is used to#pragma xmp nodes p(*)#pragma xmp template t(0:N-1)#pragma xmp distributed t(block) onto p...
double w[N], p[N], q[N], r[N];#pragma xmp align [i] with t(i):: p, q, r, w
#pragma xmp shadow [*] :: p...
/* code fragment from conj_grad in NPB CG */sum = 0.0;
#pragma xmp loop on t(j) reduction(+:sum)for (j = 1; j <= lastcol-firstcol+1; j++) {sum = sum + r[j]*r[j];}rho = sum;for (cgit = 1; cgit <= cgitmax; cgit++) {#pragma xmp reflect p#pragma xmp loop on t(j)for (j = 1; j <= lastrow-firstrow+1; j++) {sum = 0.0;
for (k = rowstr[j]; k <= rowstr[j+1]-1; k++) {sum = sum + a[k]*p[colidx[k]];}w[j] = sum;}#pragma xmp loop on t(j)for (j = 1; j <= lastcol-firstcol+1; j++) {q[j] = w[j];}…
}XcalableMP Specification Working Group
Objectives and Mission• Making a draft on XcalableMP parallel language for“standard” parallel programming.
• To propose the draft to “world-wide” community.
Members
Academia: M. Sato, T. Boku (compiler and system, U.
Tsukuba), K. Nakajima (app. and programming, U. Tokyo), T.
Nanri (system, Kyushu U.), Y. Okabe, M. Yasugi (HPF andcompiler, Kyoto U.)
Research Lab.: M. Yokokawa (RIKEN), H. Sakagami (app.
and HPF, NIFS), Y. Matsuo (app., JAXA), H. Uehara (app.,JAMSTEC/ES)
Industries: H. Iwashita and K. Hotta (HPF and XPFortran,Fujitsu), H. Murai and S. Sakon (HPF, NEC), T. Anzaki andK. Negishi (Hitachi)
specify the mapping of iteration. By using the same templateused for the data distribution, iterations are assigned to the nodeof the data. It should be noted that in XcalableMP theprogrammer must control all data reference required
computations done locally by any appropriate directives.
A fragment of code taken from NBP CG (one-dimensionalparallelization) is shown in the box. Shadow directives are usedto declare the shadow region of each array, which can besynchronized by reflect directive..
In order to describe communication, the gmove construct is a
powerful operation in global-view programming in XcalableMP: Itcopies data of a distributed array in global-view. This directive isfollowed by the assignment statement of scalar value and arraysections.
In addition to the "global view" programming described above,
XcalableMP also includes CAF-like PGAS (Partitioned Global
Address Space) feature as “local view” programming.
XcalableMP adopts coarray notations as an extension oflanguages for local view programming. In case of Fortran as thebase language, most coarray notations are compatible to that ofCoarray Fortran(CAF) expect that the task constructs are used
for task parallelism. In order to use coarray notations in C, we
propose some language extension of the language. A coarray isdeclared by the coarray directive in C.

References[1] H. Murai, T. Araki, Y. Hayashi, K. Suehiro and Y. Seo:
Implementation and Evaluation of HPF/SX V2,
Concurrency and Computation - Practice & Experience,
Vol.14, No. 8-9, Wiley (2002), 603-629.

[2] J. Lee, M. Sato and T. Boku, "OpenMPD: A
Directive-Based Data Parallel Language Extensions forDistributed Memory Systems", First International Workshop
on Parallel Programming Models and Systems Softwarefor High-End Computing (P2S2), 2008
This research is carried out as a part of “Seamless andHighly-productive Parallel Programming Environment for
High-performance computing” project funded by Ministry ofEducation, Culture, Sports, Science and Technology,JAPAN.