Highly productive
parallel programming language

XcalableMP
Masahiro NAKAO
Center for Computational Sciences
University of Tsukuba

Background
MPI is widely used as a parallel programming modelon distributed memory systems
However, writing MPI programs is often atime-consuming and a complicated process

Another programming model is needed !!
High performance
Easy to program

Development of XcalableMP
T2K Open Supercomputer Alliance@SC11

Whatʼs XcalableMP?
XcalableMP(XMP) is a new programming model andlanguage for distributed memory systems
XMP is proposed by XcalableMP Speciﬁcation WorkingGroup(XMP WG)
XMP WG is a special interest group, which is organized tomake a draft on “petascale” parallel language
XMP WG consists of members from academia(U. Tsukuba, U. Tokyo, Kyoto U. and Kyusyu U.),research labs(RIKEN, NIFS, JAXA, JAMSTEC/ES) andindustries(Fujitsu, NEC, Hitachi)

XMP Speciﬁcation version 1.0 released !!
http://www.xcalablemp.org/
T2K Open Supercomputer Alliance@SC11

AgendaOverview of XMP
XMP Programming ModelXMP Directives

T2K Open Supercomputer Alliance@SC11

AgendaOverview of XMP
XMP Programming ModelXMP Directives

T2K Open Supercomputer Alliance@SC11

Overview of XMP
XMP is a directive-based language extension like
OpenMP and HPF based on C and Fortran95
To reduce code-writing and educational costs“Scalable” for Distributed Memory ProgrammingA thread starts execution in each node
independently (as in MPI)

“Performance-aware” for explicit
communication, sync. and work-sharing

node1 node2 node3

Directives
Comm, sync and work-sharing

All actions occur when directives areencountered
All actions are taken by directives for being “easy-tounderstand” in performance tuning (diﬀerent from HPF)
T2K Open Supercomputer Alliance@SC11

XMP Code Example
int	 array[100];#pragma	 xmp	 nodes	 p(*)#pragma	 xmp	 template	 t(0:99)#pragma	 xmp	 distribute	 t(block)	 onto	 p#pragma	 xmp	 align	 array[i]	 with	 t(i)main(){#pragma	 xmp	 loop	 on	 t(i)	 reduction(+:res)	 	 	 for(i	 =	 0;	 i	 <	 100;	 i++){	 	 	 	 	 	 array[i]	 =	 func(i);	 	 	 	 	 	 res	 +=	 array[i];	 	 	 }
}datadistribution

work mapping& reduction

Programmer adds XMP directives to serial code<serial incremental parallelization>
T2K Open Supercomputer Alliance@SC11

The same code written in MPI
int	 array[100];	 
main(int	 argc,	 char	 **argv){	 	 	 	 MPI_Init(&argc,	 &argv);	 	 	 	 MPI_Comm_rank(MPI_COMM_WORLD,	 &rank);	 	 	 	 MPI_Comm_size(MPI_COMM_WORLD,	 &size);	 	 	 	 dx	 =	 100/size;	 	 	 	 	 llimit	 =	 rank	 *	 dx;	 	 	 	 if(rank	 !=	 (size	 -1))	 ulimit	 =	 llimit	 +	 dx;	 	 	 	 else	 ulimit	 =	 100;	 	 	 	 temp_res	 =	 0;	 	 	 	 for(i=llimit;	 i	 <	 ulimit;	 i++){	 	 	 	 	 	 	 	 	 	 	 	 array[i]	 =	 func(i);	 	 	 	 	 	 	 	 	 	 	 	 temp_res	 +=	 array[i];	 	 	 	 	 	 	 	 	 	 	 	 }	 	 	 	 MPI_Allreduce(&temp_res,	 &res,	 1,	 MPI_INT,	 MPI_SUM,	 MPI_COMM_WORLD);	 	 	 	 MPI_Finalize(	 );}T2K Open Supercomputer Alliance@SC11

AgendaOverview of XMP
Programming ModelXMP Directives

T2K Open Supercomputer Alliance@SC11

Programming Model
Global View Model (like as HPF)
programmer describes data distribution, work mapping andinter-node comm. with adding directives to serial code
• support typical techniques for work-mapping and data-mapping• rich communication and sync. directives, such as “shadow”,“reﬂect” and “gmove”

Local View Model (like as Co-array Fortran)
enable programmer to communicate with node index

one-sided communication using language extension(also deﬁned C)

T2K Open Supercomputer Alliance@SC11

Data Distribution
The directives specify adata distribution amongnodes#pragma	 xmp	 nodes	 p(4)#pragma	 xmp	 template	 t(0:15)#pragma	 xmp	 distribute	 t(block)	 on	 p#pragma	 xmp	 align	 array[i]	 with	 t(i)0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
array[ ]Node 0Node 1
Node 2
Node 3Distributed ArrayT2K Open Supercomputer Alliance@SC11

Parallel Execution of loop
Loop directive is inserted #pragma	 xmp	 nodes	 p(4)#pragma	 xmp	 template	 t(0:15)
before loop statement#pragma	 xmp	 distribute	 t(block)	 on	 p#pragma	 xmp	 loop	 on	 t(i)for(i=2;i<=10;i++){...}#pragma	 xmp	 align	 array[i]	 with	 t(i)0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
array[ ]Node 0Node 1

Execute “for” loop inparallel with affinity toarray distribution

Node 2
Node 3
Each node computes Red elements in parallel
T2K Open Supercomputer Alliance@SC11

shadow/reﬂect directives
Synchronize data only on shadow region
If neighbor data is required to communicate, then only shadow
area can be considered#pragma	 xmp	 shadow	 array[1:1]for(...) b[i] = array[i-1] + array[i+1];

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
array[ ]Node 0Node 1
Node 2
Node 3#pragma	 xmp	 reflect	 array

T2K Open Supercomputer Alliance@SC11

shadow/reﬂect directives
Synchronize data only on shadow region
If neighbor data is required to communicate, then only shadow
area can be considered#pragma	 xmp	 shadow	 array[1:1]for(...) b[i] = array[i-1] + array[i+1];

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
array[ ]Node 0Node 1

Reflect directive isto synchronize only
shadow regionNode 2
Node 3#pragma	 xmp	 reflect	 array

T2K Open Supercomputer Alliance@SC11

shadow/reﬂect directives
Synchronize data only on shadow region
If neighbor data is required to communicate, then only shadow
area can be considered#pragma	 xmp	 shadow	 array[1:1]for(...) b[i] = array[i-1] + array[i+1];

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
array[ ]Node 0Node 1

Reflect directive isto synchronize only
shadow region

Node 2
Node 3#pragma	 xmp	 reflect	 array

T2K Open Supercomputer Alliance@SC11

Gmove (Global View Communication)communication for distributed array
We extend “array section notation” in C
Programmer doesnʼt need to know
where each data is distributed

0 1 2 3 4 5 6 7a[ ]#pragma	 xmp	 gmovea[2:4]	 =	 b[3:4];

0 1 2 3 4 5 6 7b[ ]Node 0Node 1
Node 2
Node 3T2K Open Supercomputer Alliance@SC11

Gmove (Global View Communication)basecommunication for distributed array
We extend “array section notation” in C
Programmer doesnʼt need to know
where each data is distributed

0 1 2 3 4 5 6 7a[ ]length#pragma	 xmp	 gmove
a[2:4]	 =	 b[3:4];

0 1 2 3 4 5 6 7b[ ]Node 0Node 1
Node 2
Node 3T2K Open Supercomputer Alliance@SC11

Gmove (Global View Communication)basecommunication for distributed array
We extend “array section notation” in C
Programmer doesnʼt need to know
where each data is distributed

0 1 2 3 4 5 6 7a[ ]length#pragma	 xmp	 gmove
a[2:4]	 =	 b[3:4];

0 1 2 3 4 5 6 7b[ ]Node 0Node 1
Node 2
Node 3T2K Open Supercomputer Alliance@SC11

Gmove (Global View Communication)basecommunication for distributed array
We extend “array section notation” in C
Programmer doesnʼt need to know
where each data is distributed

0 1 2 3 4 5 6 7a[ ]length#pragma	 xmp	 gmove
a[2:4]	 =	 b[3:4];

0 1 2 3 4 5 6 7b[ ]Node 0Node 1
Node 2
Node 3T2K Open Supercomputer Alliance@SC11

Gmove (Global View Communication)basecommunication for distributed array
We extend “array section notation” in C
Programmer doesnʼt need to know
where each data is distributed

0 1 2 3 4 5 6 7a[ ]length#pragma	 xmp	 gmovea[2:4]	 =	 b[3:4];

0 1 2 3 4 5 6 7b[ ]Node 0Node 1
Node 2
Node 3T2K Open Supercomputer Alliance@SC11

XMP Global view directives
Broadcast#pragma xmp bcast (var)Reduction#pragma xmp reduction (op: var)

Barrier#pragma xmp barrier

T2K Open Supercomputer Alliance@SC11

Local View Model
We adopt Co-Array as PGAS (Partitioned Global
Address Space) feature
One-sided communication for local data(Put/Get)
High interoperability with MPI
node 1

node 2

a[10]

a[10]

b[10]

b[10]

#pragma	 xmp	 coarray	 b・・・
a[0:4]	 =	 b[3:4]:[2];indicate node number
T2K Open Supercomputer Alliance@SC11

Local View Model
We adopt Co-Array as PGAS (Partitioned Global
Address Space) feature
One-sided communication for local data(Put/Get)
High interoperability with MPI
node 1

node 2

a[10]

a[10]

b[10]

b[10]

#pragma	 xmp	 coarray	 b・・・
a[0:4]	 =	 b[3:4]:[2];indicate node number

36

T2K Open Supercomputer Alliance@SC11

Local View Model
We adopt Co-Array as PGAS (Partitioned Global
Address Space) feature
One-sided communication for local data(Put/Get)
High interoperability with MPI
node 1

node 2

a[10]

a[10]

b[10]

b[10]

#pragma	 xmp	 coarray	 b・・・
a[0:4]	 =	 b[3:4]:[2];indicate node number36

T2K Open Supercomputer Alliance@SC11

Other Functions
Easy Hybrid Programming#pragma	 xmp	 loop	 on	 t(i)	 threads
for(i=2;i<=10;i++){...}

Interface of scalasca and tlog proﬁling tools#pragma	 xmp	 gmove	 profile...
#pragma	 xmp	 loop	 on	 t(i)	 profile
T2K Open Supercomputer Alliance@SC11

More Information

http://www.xcalablemp.org/

T2K Open Supercomputer Alliance@SC11

Summary & Future workXcalableMP was proposed as a new programming model
to facilitate program parallel applications for distributed
memory systems
XcalableMP C language version compiler
http://www.xcalablemp.org
Future work
For accelerators(GPU, etc)
Parallel I/OInterface of MPI library, and so on

T2K Open Supercomputer Alliance@SC11

Thank you for your attention!!!

Q & A?
Acknowledgements:
We would like to thank XMP-WG members forvaluable discussions and comments
T2K Open Supercomputer Alliance@SC11