Highly productive
parallel programming language

XcalableMP
Masahiro NAKAO
Center for Computational Sciences
University of Tsukuba

Background
MPI is widely used as a parallel programming modelon distributed memory systems
However, writing MPI programs is often atime-consuming and complicated process

Another programming model is needed !!
High performance
Easy to program

Development of XcalableMP
T2K Open Supercomputer Alliance@SC10

Whatʼs XcalableMP?
XcalableMP(XMP) is a new programming model andlanguage for distributed memory systems
XMP is proposed by XcalableMP Speciﬁcation WorkingGroup(XMP WG)
XMP WG is a special interest group, which is organized tomake a draft on “petascale” parallel language
XMP WG consists of members from academia(U. Tsukuba, U. Tokyo, Kyoto U. and Kyusyu U.),research labs(RIKEN, NIFS, JAXA, JAMSTEC/ES) andindustries(Fujitsu, NEC, Hitachi)

XMP was selected as a ﬁnalist of HPCC Class2 in SC10
and SC09
T2K Open Supercomputer Alliance@SC10

AgendaOverview of XMP
Programming ModelXMP Directives

T2K Open Supercomputer Alliance@SC10

Overview of XMP
XMP is a directive-based language extension like
OpenMP and HPF based on C and Fortran95
To reduce code-writing and educational costs“Scalable” for Distributed Memory ProgrammingA thread starts execution in each node
independently (as in MPI)

“Performance-aware” for explicit
communication, sync. and work-sharing

node1 node2 node3

Directives
Comm, sync and work-sharing

All actions occur when directives areencountered
All actions are taken by directives for being “easy-tounderstand” in performance tuning (diﬀerent from HPF)
T2K Open Supercomputer Alliance@SC10

XMP Code Example
int	 array[100];#pragma	 xmp	 nodes	 p(*)#pragma	 xmp	 template	 t(0:99)#pragma	 xmp	 distribute	 t(block)	 onto	 p#pragma	 xmp	 align	 array[i]	 with	 t(i)main(){#pragma	 xmp	 loop	 on	 t(i)	 reduction(+:res)	 	 	 for(i	 =	 0;	 i	 <	 100;	 i++){	 	 	 	 	 	 array[i]	 =	 func(i);	 	 	 	 	 	 res	 +=	 array[i];	 	 	 }
}datadistribution

work sharing& reduction

Programmer adds XMP directives to serial code<serial incremental parallelization>
T2K Open Supercomputer Alliance@SC10

The same code written in MPI
int	 array[100];	 
main(int	 argc,	 char	 **argv){	 	 	 	 MPI̲Init(&argc,	 &argv);	 	 	 	 MPI̲Comm̲rank(MPI̲COMM̲WORLD,	 &rank);	 	 	 	 MPI̲Comm̲size(MPI̲COMM̲WORLD,	 &size);	 	 	 	 dx	 =	 100/size;	 	 	 	 	 llimit	 =	 rank	 *	 dx;	 	 	 	 if(rank	 !=	 (size	 -1))	 ulimit	 =	 llimit	 +	 dx;	 	 	 	 else	 ulimit	 =	 100;	 	 	 	 temp̲res	 =	 0;	 	 	 	 for(i=llimit;	 i	 <	 ulimit;	 i++){	 	 	 	 	 	 	 	 	 	 	 	 array[i]	 =	 func(i);	 	 	 	 	 	 	 	 	 	 	 	 temp̲res	 +=	 array[i];	 	 	 	 	 	 	 	 	 	 	 	 }	 	 	 	 MPI̲Allreduce(&temp̲res,	 &res,	 1,	 MPI̲INT,	 MPI̲SUM,	 MPI̲COMM̲WORLD);	 	 	 	 MPI̲Finalize(	 );}T2K Open Supercomputer Alliance@SC10

AgendaOverview of XMP
Programming ModelXMP Directives

T2K Open Supercomputer Alliance@SC10

Programming Model
Global View Model(like as HPF)
programmer describes data distribution, work mapping andinter-node comm. with adding directives to serial code
• support typical techniques for data/task parallelization• rich communication and sync. directives, such as “shadow”,“reﬂect” and “gmove”

Local View Model(like as Co-array Fortran)
one-sided communication using language extension(also deﬁned C)
enable programmer to communicate easily among nodes

T2K Open Supercomputer Alliance@SC10

Data Distribution
The directives specify adata distribution amongnodes#pragma	 xmp	 nodes	 p(4)#pragma	 xmp	 template	 t(0:15)#pragma	 xmp	 distribute	 t(block)	 on	 p#pragma	 xmp	 align	 array[i]	 with	 t(i)0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
array[ ]Node 0Node 1
Node 2
Node 3Distributed ArrayT2K Open Supercomputer Alliance@SC10

Parallel Execution of loop
Loop directive is inserted #pragma	 xmp	 nodes	 p(4)#pragma	 xmp	 template	 t(0:15)
before loop statement#pragma	 xmp	 distribute	 t(block)	 on	 p#pragma	 xmp	 loop	 on	 t(i)for(i=2;i<=10;i++){...}#pragma	 xmp	 align	 array[i]	 with	 t(i)0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15array[ ]Node 0Node 1

Execute “for” loop inparallel with affinity toarray distribution

Node 2
Node 3
Each node computes Red elements in parallel
T2K Open Supercomputer Alliance@SC10

shadow/reﬂect directives
Exchange data/Sync. only on shadow region
If neighbor data is required to communicate, then only shadow
area can be considered#pragma	 xmp	 shadow	 array[1:1]for(...) b[i] = array[i-1] + array[i+1];

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15array[ ]Node 0Node 1Reflect directive isto synchronize onlyshadow region

Node 2
Node 3#pragma	 xmp	 reflect	 array

T2K Open Supercomputer Alliance@SC10

Gmove (Global View Communication)communication for distributed array
We extend “array section notation” in C
Programmer doesnʼt need to know
where each data is distributed

0 1 2 3 4 5 6 7a[ ]#pragma	 xmp	 gmove
a[2:4]	 =	 b[3:6];

0 1 2 3 4 5 6 7b[ ]Node 0Node 1
Node 2
Node 3T2K Open Supercomputer Alliance@SC10

XMP Global view directivesBroadcast#pragma xmp bcast (var)Reduction#pragma xmp reduction (op: var)

Barrier#pragma xmp barrier

Task parallelism#pragma xmp task on node-set

T2K Open Supercomputer Alliance@SC10

Local View Model
We adopt Co-Array as PGAS (Partitioned Global
Address Space) feature
One-sided communication for local data(Put/Get)
High interoperability with MPI
node 1

node 2

a[10]

a[10]

b[10]

b[10]

#pragma	 xmp	 coarray	 b・・・
a[0:3]	 =	 b[3:6]:[2];indicate node number

36

T2K Open Supercomputer Alliance@SC10

More Information

http://www.xcalablemp.org/

T2K Open Supercomputer Alliance@SC10

Summary & Future work
XcalableMP was proposed as a new programming model
to facilitate program parallel applications for distributed
memory systems
XcalableMP C language version compiler
http://www.xcalablemp.org
Future work(Issues under discussion)Multicores (SMP) Cluster and Hybrid programming with
OpenMP
Parallel I/O
Extension to GPGPU, Fault tolerant, Others

T2K Open Supercomputer Alliance@SC10

Thank you for your attention!!!

Q & A?
Acknowledgements:
We would like to thank XMP-WG members forvaluable discussions and comments
T2K Open Supercomputer Alliance@SC10