Productivity and Performance ofthe HPC Challenge Benchmarks withthe XcalableMP PGAS Language
Masahiro Nakao, Hitoshi Murai,Takenori Shimosaka, Mitsuhisa Sato
Center for Computational Sciences, University of Tsukuba, JapanRIKEN Advanced Institute for Computational Science, Japan

7th International Conference on PGAS Programming Models@Edinburgh

Overview of XcalableMP（XMP）
Directive-based language extension of C99 and Fortran2008
The same directives are used in XMP/C and XMP/Fortran
Coarray syntax is available in XMP/C and XMP/Fortran

XMP/CXMP/Fortranint array[16];#pragma xmp nodes p(4)#pragma xmp template t(0:15)#pragma xmp distribute t(block) onto p#pragma xmp align array[i] with t(i)integer array(16);!$xmp nodes p(4)!$xmp template t(1:16)!$xmp distribute t(block) onto p!$xmp align array(i) with t(i)main(){...
#pragma xmp loop on t(i)for(i = 0; i < 16; i++){array[i] = func(i);}}program main...
!$xmp loop on t(i)do i=1,16
array(i) = func(i)done
end program2

Overview of XcalableMP（XMP）
Directive-based language extension of C99 and Fortran2008
The same directives are used in XMP/C and XMP/Fortran
Coarray syntax is available in XMP/C and XMP/Fortran

XMP/CXMP/Fortranint b[10]:[*];

integer b(10)[*]if(me == 1){b[0:5]:[2] = b[0:5]; // Put}if(me == 1) thenb(1:5)[2] = b(1:5) // Put
end if

XMP/Fortran is upward
compatible with the Fortran2008

3

Objective
Examine eﬀectiveness of designs of the XMP PGAS language
for improved productivity and performance of HPC systems
Evaluate the productivity and the performance of XMP throughimplementations of the HPC Challenge (HPCC) Benchmarks
Use 32,768 compute nodes at a maximum on the K computers (whichconsists of 88,128 compute nodes)

ranked 1st in the Top500 onJune, 2011
4

Agenda1. Introduce XMP features- Global-view memory model with XMP directives- Local-view memory model with coarray syntax- Designs of XMP for HPC applications
2. Explain implementations of the HPCC Benchmarks andevaluate their productivity and performance
3. Discuss experimental results
4. Summarize our presentation

5

XMP Global-view model (1/3)
The directives specify adata distribution among
nodes

int	 array[16];#pragma	 xmp	 nodes	 p(4)#pragma	 xmp	 template	 t(0:15)#pragma	 xmp	 distribute	 t(block)	 on	 p#pragma	 xmp	 align	 array[i]	 with	 t(i)0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
array[16]Node 1
Node 2
Node 3
Node 4
Distributed Array
6

XMP Global-view model (2/3)
Loop directive is toparallelize loop statement#pragma	 xmp	 loop	 on	 t(i)for(i=2;i<=10;i++){...}

array[16]Node 1
Node 2

int	 array[16];#pragma	 xmp	 nodes	 p(4)#pragma	 xmp	 template	 t(0:15)#pragma	 xmp	 distribute	 t(block)	 on	 p#pragma	 xmp	 align	 array[i]	 with	 t(i)0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Execute “for” loop inparallel with affinity toarray distribution

Node 3
Node 4Each node computes Red elements in parallel
7

XMP Global-view model (3/3)
Data communication directives : broadcast, reduction, gmove
gmove directive[start_index : length]

Transfer data while keeping the global image
by using "array section notation"

0 1 2 3 4 5 6 7a[10 ]#pragma	 xmp	 gmovea[2:4]	 =	 b[3:4];

0 1 2 3 4 5 6 7b[10 ]Node 1
Node 2
Node 3
Node 48XMP Local-view model
Support coarray syntax in XMP/C and XMP/Fortran
XMP/Fortran is upward compatible with the Fortran 2008
XMP/C also uses array section notation in coarray syntax

XMP/C
int b[10]:[*];

node 1

node 2

a[10]

a[10]

b[10]

b[10]

// Declare:if(me == 1)a[0:3] = b[4:3]:[2];// Get

The node1 gets b[4:3] in the node 246

It is easy to express one-sided communication for local data (Put/Get).
Can mix XMP global-view directives with coarray-syntax.

9Designs of XMP for HPC applications
PGAS programming language must have both high productivity
and high performance
The productivity of HPC applications consists of programming cost,educational cost, porting cost, and tuning cost

Designs of XMP for HPC applications (1/3)
Easy writing of various parallel applications <programming cost↓>[Global-view] Enable parallelization of an original sequential code using
minimal modiﬁcation with simple directives[Local-view] Easy to express one-sided comm. with coarray-syntax
Easy learning <educational cost↓>
Extension of C and Fortran10

Designs of XMP for HPC applications
Design of XMP for HPC applications (2/3)
Numerical libraries (BLAS etc.) & MPI library can be invoked from XMP
program <porting↓, tuning cost↓, performance↑>
int	 array[16];#pragma	 xmp	 nodes	 p(4)#pragma	 xmp	 template	 t(0:15)#pragma	 xmp	 distribute	 t(block)	 onto	 p#pragma	 xmp	 align	 array[i]	 with	 t(i)main(){	 	 	 ...	 	 	 cblas_dgemm(..,	 &array[k],	 ...);}

This is a code example wherea global array is used inBLAS library.
a pointer of a global arrayindicates a local pointer on thenode to which it is distributed

XMP inquiry functions obtain local memory information from a global array.
For example, xmp_array_lead_dim() obtains a local leading dimension ofa global array.

11

Designs of XMP for HPC applications
Design of XMP for HPC applications (3/3)"OpenMP-safe", except for comm. directives <performance↑>
Programmer can use OpenMP directives in XMP

XMP/C

XMP/Fortranint array[16];#pragma xmp nodes p(4)#pragma xmp template t(0:15)#pragma xmp distribute t(block) onto p#pragma xmp align array[i] with t(i)integer array(16);!$xmp nodes p(4)!$xmp template t(1:16)!$xmp distribute t(block) onto p!$xmp align array(i) with t(i)main(){...
#pragma xmp loop on t(i)#pragma omp parallel for
for(i = 0; i < 16; i++){array[i] = func(i);}}program main...
!$xmp loop on t(i)!$omp parallel dodo i=1,16
array(i) = func(i)done
end program

12Agenda1. Introduce XMP features- Global-view memory model with XMP directives- Local-view memory model with coarray syntax- Designs of XMP for HPC applications
2. Explain implementations of the HPCC Benchmarks andevaluate their productivity and performance
3. Discuss experimental results
4. Summarize our presentation

13

HPC Challenge(HPCC) Benchmarks
The HPCC Benchmarks are a set of benchmarks to evaluate
multiple attributes on an HPC system
The HPCC Benchmarks are also used at HPCC Award Competition
at Supercomputer Conference
In Class 1, only the performance of an HPC system is evaluated
In Class 2, the productivity and performance of a programming language
are evaluatedbased on hpcc-1.4 written in
RandomAccessC + Fortran + MPI which isHigh Performance Linpack (HPL)
released by the HPCC
community (http://Fast Fourier Transform (FFT)
icl.cs.utk.edu/hpcc/software/)
STREAM
weak scaling

14

Evaluation
Omni XMP Compiler version 0.7-alphaReference Implementation
Open Source

http://www.hpcs.cs.tsukuba.ac.jp/omni-compiler/xcalablemp/

Optimized for the K computer"./conﬁgure --target=Kcomputer-linux-gnu"
To use high-speed one-sided communication on the K computer,the coarray syntax is translated into calling the extended RDMAThis Compiler will be released in Nov. 201315Environment
The K computer

HA-PACSCPUSPARC64 VIIIfx 2.0GHz
8Cores, 128GFlops

Xeon E5-2670 2.6GHz x2
8Cores x2, 332.8GFlops

MemoryDDR3 SDRAM 16GB
64GB/s/Socket

DDR3 SDRAM 128GB
51.4GB/s/Socket

Network

Torus fusion six-dimensional
mesh/torus network, 5GB/s

Infiniband QDRx2rails
Fat-tree network, 4GB/s

HA-PACS has GPUs
as an accelerator.
But we used only CPU.

To measure the performance, we used 32,768 nodes at a maximumof the K computer and 64 nodes at a maximum of HA-PACS
16

RandomAccess
The RandomAccess benchmark measures the performance ofrandom integer updates of memory via interconnect
Each process randomly updates table of other processes
It is suitable to use coarray syntax
To reduce communication times, our algorithm is iterated over sets ofCHUNK updates on each node
Our algorithm is almost the same as the hpcc-1.4 RandomAccess17

RandomAccess
Source lines of code (SLOC) is 258, written in XMP/Cu64Int recv[MAXLOGPROCS][RCHUNK+1]:[*];
for(...){...
send[isend][0] = nsend; // set "number of transfer elements"
recv[j][0:nsend+1]:[send_target] = send[isend][0:nsend+1];#pragma xmp sync_memory#pragma xmp post(p(send_target), 0)...
#pragma xmp wait(p(recv_target))#pragma xmp sync_memory
nrecv = recv[j-1][0];sort_data(&recv[j-1][1], nrecv, ..);...
}

Declare coarray

PUT
Ensure to finish
PUT operation

18Performance of RandomAccess
The modified hpcc-1.4 RandomAccess, for which thefunctions updating the table are specifically optimized
for the K computer
163GUPS in 16,384 nodes(131,072 CPU Cores)

GUPS (Giga UPdates per Second)ﬂat-MPI (8 Process/Node)
Good Performance !!

19High Performance Linpack (HPL)
HPL measures the ﬂoating point rate of execution to solve a
dense system of linear equations using LU factorization
In our implementation, the coeﬃcient matrix is distributed in blockcyclic manner like hpcc-1.4 HPLThis distribution is useful to perform good load balance
BLAS Library is used
A[N][N]
NB

node 1
node 2・・double	  A[N][N];#pragma	  xmp	  nodes	  p(4,2)#pragma	  xmp	  template	  t(0:N-­‐1,	  0:N-­‐1)#pragma	  xmp	  distribute	  t(cyclic(NB),	  \	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  cyclic(NB))	  onto	  p#pragma	  xmp	  align	  A[i][j]	  with	  t(j,i)

node 8NBHigh Performance Linpack (HPL)
Panel Broadcast by using gmove directive
double	  A_L[N][NB];#pragma	  xmp	  align	  L[i][*]	  with	  t(*,i)
	  	  	  	  :#pragma	  xmp	  gmove
L[k:len][0:NB]	  =	  A[k:len][j:NB];

klenSLOC is 288, written in XMP/C

21

Performance of HPL
8 Threads/Process on 1 node

543 TFlops in 8,192 nodes(65,536 CPU Cores),
53% of the theoretical peak

100 GFlops in 1 node, 8 CPU Cores,
78% of the theoretical peak

In spite of weak scaling, the paralleleﬃciency is not very good.
22

Fast Fourier Transform (FFT)FFT measures the ﬂoating point rate of execution for doubleprecision complex one-dimensional Discrete Fourier Transform
We parallelized only a subroutine “PZFFT1D0”, which is the mainkernel of the hpcc-1.4 FFT

23

Fast Fourier Transform (FFT)Matrix transposition is implemented by using gmove directive
The SLOC of PZFFT1D0 is 65, written in XMP/Fortran + OpenMP!$XMP	  distribute	  tx(block)	  onto	  p!$XMP	  distribute	  ty(block)	  onto	  p!$XMP	  align	  A(*,i)	  with	  ty(i)!$XMP	  align	  A_WORK(i,*)	  with	  tx(i)!$XMP	  align	  B(*,i)	  with	  tx(i)	  	  	  :!$XMP	  gmove
A_WORK(:,:)	  =	  A(:,:)	  !	  all-­‐to-­‐all!$XMP	  loop	  on	  tx(I)!$OMP	  parallel	  doDO	  60	  I=1,NX	  	  DO	  70	  J=1,NY	  	  	  	  B(J,I)=A_WORK(I,J)
	  	  60	  CONTINUE
70	  CONTINUE!$XMP	  gmove

1. Node 2 transfers data to node 1with packing it
2. Node 1 copies A_WORK() toB() by using XMP and OpenMPdirectives

24Performance of FFT
8 Thread/process on 1 node
24TFlops in 32,768 nodes(262,144 CPU Cores)
50% of performance of thehpcc-1.4 FFT

25

Agenda1. Introduce XMP features- Global-view memory model with XMP directives- Local-view memory model with coarray syntax- Designs of XMP for HPC applications
2. Explain implementations of the HPCC Benchmarks andevaluate their productivity and performance
3. Discuss experimental results
4. Summarize our presentation

26

Comparison with hpcc-1.4 (MPI)
Productivity

hpcc-1.4 XMP

RandomAccess : SLOC : 938 -> 258
coarray is a more convenient to express communications
HPL : SLOC : 8,800 -> 288
PZFFT1D0 of FFT : SLOC : 101 -> 65
XMP global view enables programmers to develop parallel
applications easily

Performance
RandomAccess : Good !
HPL and FFT : The performances of XMP implementations are worse
than those of hpcc-1.427

Discussion (2/3)
Overhead of gmove directive[HPL] Gmove directive is a blocking
operation. Communication andcomputation are not overlapped.
#pragma	  xmp	  gmove
A_L[k:len][0:NB]	  =	  A[k:len][j:NB];[FFT] In gmove directive, data
pack/unpack operation is not executedwith thread-parallelization!$XMP	  gmove
A_WORK(:,:)	  =	  A(:,:)

28

Discussion (3/3)
To improve performancenon-blocking gmove operation#pragma	  xmp	  gmove	  async(async-id)
A_L[k:len][0:NB]	  =	  A[k:len][j:NB];

data pack/unpack with threadedparallelization in gmove(overlapped	  computation)#pragma	  xmp	  wait_async	  async(async-id)

Improving the performance of the gmove is important. But, ...
While level of abstraction of the gmove is very high, the performanceof the gmove remains unclarity
Gmove improves the productivity, but may become worse theperformance
If the performance of the gmove has a problem, we recommend that
programmer will be able to rewrite the communication with coarraysyntax or MPI library29

Summary
Examine the eﬀectiveness of designs of XMP for improved theproductivity and the performance of a HPC system
Global-view model and Local-view modelCan use Numerical Library with XMP inquiry functions

Evaluate the productivity and the performance throughimplementations of HPCC Benchmarks on the K computer
Good productivity and performance in 32,768 nodes at a maximum
But the gmove directive has scope to continue to improve

Future work
Support non-blocking operation and thread-parallelization
Retry to evaluate their performances for next HPCC Award at SC1330