Productivity and Performance of
Global-View Programming with
XcalableMP PGAS Language◯Masahiro Nakao, Jinpil Lee,Taisuke Boku, Mitsuhisa Sato
Center for Computational Sciences, University of Tsukuba
RIKEN Advanced Institute for Computational Science(AICS)

Background
Partitioned Global Address Space (PGAS) model
has been proposed
Global address space where any processes
can access distributed data transparently
Increases development productivityof parallel applications
The global address space is logicallypartitioned between the processes
Enables programmers to performperformance-aware parallel programming
Two kinds of memory abstract model :Global-view model, Local-view model
2

CCGrid2012@Ottawa, Canada

Overview of XcalableMP
XcalableMP(XMP) is a PGAS language http://www.xcalablemp.org
Directive-based extension of C99 and Fortran95“Performance-aware” parallel programming (after slide)The basic execution model is SPMD
Two memory abstract models in one language :Global-view model
Local-view model (compatible with the coarray Fortran)#pragma	 xmp	 loop	 on	 t(i)	 reduction(+:res)for(i	 =	 0;	 i	 <	 100;	 i++){	 	 	 array[i]	 =	 func(i);	 	 	 res	 +=	 array[i];}3CCGrid2012@Ottawa, CanadaObjective
XMP global-view model is useful when parallelizing dataparallel programs with minimum code modiﬁcation
Consider the Productivity and Performance ofXMP global-view modelHow ?
Compare XMP with other PGAS Language
Global-view model

Uniﬁed Parallel C (UPC)
Global Arrays (GA)

4

Why ?

C language extension

Coarray Fortran (CAF)

SPMD

ChapelMany people use

X10
CCGrid2012@Ottawa, Canada

Outline
Summarize features of XMP and UPC in global-view
model
Evaluate their Performance and Productivity throughsome benchmarks

5

CCGrid2012@Ottawa, Canada

Whatʼs Global-view model ?
Data-mapping and work-mapping automatically
Example of data-mapping :0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15a[ ]Node #1
Node #2
Node #3
Node #4
Distributed Array
6CCGrid2012@Ottawa, Canada

Whatʼs Global-view model ?
Data-mapping and work-mapping automatically
Example of work-mapping :0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15a[ ]Node #1
Node #2

Execute “for” loop inparallel with affinity toarray distribution

Node #3
Node #4
Each node computes Red elements in parallel
7CCGrid2012@Ottawa, CanadaConcepts of XMP and UPC
UPC : Distributed Shared Memory Programming
XMP : Performance-aware Programminga[i] = tmp;// a[] is a distributed array, tmp is a local variable

UPC calculates where a[i] is located and its oﬀset
XMP accesses a[i] directly (no communication)
In XMP, when accessing distributed array with communication,
XMP directive should be inserted before the access.
#pragma xmp gmove
a[i] = tmp;8

Because of this policy, XMP may access faster than UPC
CCGrid2012@Ottawa, CanadaAdvantage?
XMP implementation is very simple, but programmer
must consider whether needs communication or not
However, communication points of XMP are more
explicit than those of UPC

XMP
a[i] = tmp;

This line must not occur communication#pragma xmp gmove
a[i] = tmp;

UPC

This line may occur communicationa[i] = tmp;
9CCGrid2012@Ottawa, Canada

Access speed to distributed array
Evaluate access speed to distributed array, which has anaﬃnity with own process
Distributed array is accessed in parallel application
Access speed is important for its performancea[]
How much overhead
for internal operation?

#110#2#3#4
CCGrid2012@Ottawa, Canada

Evaluation of access speed
Read/write access speed to distributed array within
a single node (no-communication)
Type array : double
Number of elements : 2^20 (= 1M) every nodeDistribution manner : block, cyclic, block-cyclic

Tsukuba Omni XcalableMP Compiler 0.5.4 (TXMP)
Berkley Uniﬁed Parallel C 2.14.0 (BUPC)

11XMP

UPC

#pragma xmp loop on t(i)for(i = 0; i < N; i++)a[i] = tmp; // tmp is a local

upc_forall(i = 0; i < N; i++; &a[i])a[i] = tmp; // tmp is a local
CCGrid2012@Ottawa, Canada

Environment
T2K Tsukuba System : Linux cluster
CPU : AMD Opteron Quad-Core 8356 2.3GHz (4 sockets)
Memory : DDR2 667MHz 32GB
Network : Inﬁniband DDR(4rails) 8GB/s

12CCGrid2012@Ottawa, Canada

Result
GCC

TXMP-Block

TXMP-Cyclic

TXMP-Block-cyclic

BUPC-Block

BUPC-Cyclic

BUPC-Block-cyclic

BUPC

BUPC

Time（sec.）

0.020
0.015
0.010

TXMP

TXMP

Good0.005
0read
13

write
CCGrid2012@Ottawa, Canada

Discussion
UPC has a “privatization” technique to speed up foraccess to distributed array
Direct access by using a local address of a distributed array
shared double a[SIZE];
double *a_ptr;
a_ptr = &a[MYTHREAD];:
for(i=0;i<SIZE/THREADS; i++)a_ptr[i] = ....

assign a beginning address of
distributed array to local pointerBut, program is more complex,
because work-mapping must
be written by users

XMP can access to distributed array as fast as
a backend compiler without “privatization” technique
14

CCGrid2012@Ottawa, CanadaOutline
Summarize features of XMP and UPC in global-view
model
Evaluate their Productivity and Performance throughsome benchmarks

15

CCGrid2012@Ottawa, Canada

Data layout
Data layout is important toReduce communication and balance CPU loads on each node
Adjust any applications

Need to support various data layouts

16

CCGrid2012@Ottawa, Canada

UPC data distribution
UPC :shared [NB] double a[N]; // NB is a block size
block

NB = N/4

#1

#2

#3

#4

cyclic

NB = 1

blockcyclic

NB = N/8

Merit : Very easy to understandDemerit : Only in order of its memory (restriction ofmulti-dimensional array)
17

CCGrid2012@Ottawa, CanadaXMP data distribution
The directives specify a double	 a[N];data distribution among #pragma	 xmp	 nodes	 p(4)#pragma	 xmp	 template	 t(0:N-1)
nodes (inherit from HPF) #pragma	 xmp	 distribute	 t(block)	 on	 p#pragma	 xmp	 align	 a[i]	 with	 t(i)blockMulti-dimensional array is supported
double	 a[N][N];#pragma	 xmp	 nodes	 p(2,	 2)#pragma	 xmp	 template	 t(0:N-1,	 0:N-1)#pragma	 xmp	 distribute	 t(block,	 block)	 on	 p#pragma	 xmp	 align	 a[i][j]	 with	 t(i,j)
18

#1

#2

#3

#4

CCGrid2012@Ottawa, Canada

Shadow/Reﬂect directives
Supports shadow region for stencil applications
double a[9];#pragma xmp nodes p(3)#pragma xmp template t(0:9)
Data#pragma xmp distribute t(block) onto p
distribution#pragma xmp align a[i] with t(i)#pragma xmp shadow a[1:1]// set width of shadow region://changing a[]#pragma xmp reﬂect (a)

a[9]

01

#1
19

2

3

// synchronize shadow region

2 3 4 5 6

#2567

8

#3
CCGrid2012@Ottawa, Canada

Shadow/Reﬂect directives
Supports shadow region for stencil applications
double a[9];#pragma xmp nodes p(3)#pragma xmp template t(0:9)
Data#pragma xmp distribute t(block) onto pdistribution#pragma xmp align a[i] with t(i)#pragma xmp shadow a[1:1]// set width of shadow region://changing a[]#pragma xmp reﬂect (a)

a[9]

01

#120

23// synchronize shadow region

2 3 4 5 6#25678#3
CCGrid2012@Ottawa, Canada

Shadow/Reﬂect directives
Supports shadow region for stencil applications
double a[9];#pragma xmp nodes p(3)#pragma xmp template t(0:9)
Data#pragma xmp distribute t(block) onto p
distribution#pragma xmp align a[i] with t(i)#pragma xmp shadow a[1:1]// set width of shadow region://changing a[]#pragma xmp reﬂect (a)

a[9]

01

#1
21

2

3// synchronize shadow region

2 3 4 5 6#25678#3
CCGrid2012@Ottawa, CanadaLaplace Solver
To evaluate the XMP shadow function
b[y][x] = (a[y+1][x]+a[y-1][x]+a[y][x+1]+a[y][x-1])/4;a[][] and b[][] are distributed array#pragma xmp shadow a[1:1][0] // define shadow:#pragma xmp reflect (a) // synchronize shadow region#pragma xmp loop on t(y)for(y = 1; y < N-1; y++)for(x = 1; x < N-1; x++)b[y][x] = (a[y-1][x]+a[y+1][x]+a[y][x-1]+a[y][x+1])/4;

a[N][N]

This XMP code is similar to serial one.
22

CCGrid2012@Ottawa, CanadaLaplace in UPC
In UPC, we use upc̲memget() to get shadow region
if(THREADS != 1){
if(MYTHREAD == 0){
upc_memget(&bottom[1], &b[WIDTH][1], (N-2)*sizeof(double));} else if(MYTHREAD == THREADS-1){:Needs many if-else}statements to}communicate andupc_barrier;calculate shadow region
upc_forall(y=1; y<N-1; y++; &b[y][0]){if(MYTHREAD == 0){if(y == WIDTH-1){
for(x=1; x<N-1; x++) b[y][x] = (a[y-1][x] + bottom[x] + a[y][x-1] + a[y][x+1])/4;} else {:We implemented UPC-privatization version too23CCGrid2012@Ottawa, CanadaMeasurement of productivity
To measure productivity, we use a Delta SLOC metric[1]

The metric indicates how many lines of code change from theoriginal code. How many lines have been modiﬁed, added, anddeleted from the original code
The smaller the total of three metrics or the total source code
is, the better the productivity is
For example :Originalfor(i=0;i<100;i++)a[i] = func(i);24#pragma xmp loop on t(i)for(i=0;i<100;i++)a[i] = func(i);XMP
Added line : +1

upc_forall(i=0;i<100;i++;&a[i]) UPC
Modified line : +1a[i] = func(i);[1] Andrew I. et. al. , “Evaluating Coarray Fortran with the CGPOP Miniapp”,
PGAS11, 2011
CCGrid2012@Ottawa, Canada

Productivity

25

Original

TXMP

BUPC

BUPCprivatization

Total source
code34457370

Modified-

042Added-

113941

Deleted-005Total delta
SLOC-

114348CCGrid2012@Ottawa, Canada

Productivity

26

Original

TXMP

BUPC

BUPCprivatization

Total source
code

34

4573

70

Modified-

042Added-

113941

Deleted-005Total delta
SLOC-114348

Especially, the value of “Modified” and “Deleted” are 0 !!
This means XMP can parallelize it very simply
CCGrid2012@Ottawa, Canada

Productivity

27

OriginalTXMP

BUPC

BUPCprivatization

Total source
code

34

45

73

70

Modified-

042

Added-

1139

41

Deleted-005

Total delta
SLOC-

1143

48

UPC must use many “if-else” statements for Comm. andCalculation. The productivity of XMP is higher than those of UPCs
CCGrid2012@Ottawa, CanadaPerformance
Using array size is 1024x1024, (Strong scaling)
Performance(GFlops)

50

TXMP
BUPC-privatization40BUPC

30Good20
1001248163264128256

Number of CPU Cores
Performance of TXMP is higher than those of BUPCs
28

because there are many “if-else” statements in BUPCs

CCGrid2012@Ottawa, Canada

Conjugate Gradient(CG)To evaluate a more general benchmark
Need to communicate between distributed arrays andlocal variables for reduction or transposition
In XMP, we have developed by using 2D process gridand array w[], q[], r[], p[], z[] are distributed
Automatically work-mapping

In UPC, we have used UPC-CG developed by the GWUHigh-Performance Computing Lab.
Only array w[] is distributed
Manually work-mapping
http://threads.hpcl.gwu.edu/sites/npb-upc
29

CCGrid2012@Ottawa, Canada

Conjugate Gradient(CG)XMP :When the number of processes is not power-of-two,
2, 8, 32, 128, ...
Transferred data is larger than UPC-CG because unused
data is reduced by using XMP global-view communication
directive

UPC :Only used data is reduced anytime
Each thread calculates beginning and end point oftransferred data (is similar to NASA version CG)
However, the value of total delta SLOC and total sourcecode are larger than those of XMP-CG30

CCGrid2012@Ottawa, Canada

Productivity result
Implementations of XMP and UPC are based on
C language serial CG developed by RWCP in Japan

31

Original

TXMP

BUPC

BUPCprivatization

Total source
code

376

466

664

651

Modified-20

103

Added-

116

296

303

Deleted-

26

28

28

Total delta
SLOC-

162334334

CCGrid2012@Ottawa, Canada

Productivity result
Implementations of XMP and UPC are based on
C language serial CG developed by RWCP in Japan

32

Original

TXMP

BUPC

BUPCprivatization

Total source
code

376

466

664

651

Modified-20

103Added-

116

296303

Deleted-262828

Total delta
SLOC-162334

334

CCGrid2012@Ottawa, CanadaPerformance result
Size of array is 150000 x 15000 (Class C), Strong scaling

Performance(Mops)

15000

TXMP
BUPC-privatization

12000

BUPC

22%

9000

Good
6000
3000
01

2

4

8

16

32

64

128

256

Number of CPU Cores
33

CCGrid2012@Ottawa, Canada

Summary and Future work
Summary
We investigated productivity and performance of the XMP inglobal-view model to compare with the UPC
XMP supports more data layouts, and has a higher performanceaccess speed to distributed array without “privatization”
In laplace solver, the performance and productivity of XMP are
higher because XMP supports shadow region
In CG, the performance of XMP and UPC is almost the same
except 128 CPU cores, the productivity of XMP is high
XMP has a rich global-view programming model thatallows it to develop applications with a smaller cost

Future work
Evaluation for real applications in larger number of nodes
34

compare against Chapel, X10, and so on

CCGrid2012@Ottawa, Canada