2013 HPC Challenge Class 2 Submission
XcalableMP for Productivity and Performance†Masahiro Nakao, Hitoshi Murai‡

Takenori Shimosaka,‡ Mitsuhisa Sato†‡† Center for Computational Sciences, University of Tsukuba, Japan‡ RIKEN Advanced Institute for Computational Science, Japan

HPC Challenge Class II BoF@SC13, Nov. 21th

Outline
1.Whatʼs XcalableMP ?
2.Implementations
1. HPL2. RandomAccess3. FFT
4. STREAM
5. HIMENO Benchmark (Optional)

3.Performances
4.Conclusion2What is XcalableMP?
XcalableMP (XMP)
Directive-based language extension of Fortran 2008 and C99
XMP/Fortran language and XMP/C language exist (same directives)
Global-view and Local-view memory models
XMP global-view model enables parallelizing the original sequential codeusing minimal modiﬁcation with simple directives
Coarray syntax is available in both languages
XMP/Fortran is upward compatible with Fortran 2008
XMP specification ver. 1.2 at http://www.xcalablemp.org
XMP specification working group consists of members from academia,
research labs, and industries
Univ. of Tsukuba, RIKEN AICS, Fujitsu, NEC, Hitachi, and so on
Omni XMP Compiler ver. 0.7 at http://www.hpcs.cs.tsukuba.ac.jp/omni-compiler/
xcalablemp/3Benchmarks
HPCC benchmarks : HPL, RandomAccess, FFT, STREAM
Optional benchmark : Himeno benchmark
Typical stencil application (http://accc.riken.jp/2444.htm)
Evaluates performance of incompressible ﬂuid analysis code
Why do we select the HIMENO benchmark ?
To demonstrate parallelization byXMP directives which synchronizes
the overlapped regions4

Benchmarks

You can download these implementations at
http://www.hpcs.cs.tsukuba.ac.jp/omni-compiler/
xcalablemp/
5The K computer
SPARC64 VIIIfx Chip128 GFlops (2GHz x 8Cores x 8)
6MB L2 shared cache
58W
DDR3 SDRAM
16GB
64GB/sTofu Interconnect
6D mesh/torus network
5GB/s x 4links x 282,944 compute nodes (+ 5,184 IO nodes)
http://www.aics.riken.jp

6Outline
1.Whatʼs XcalableMP ?
2.Implementations
1. HPL2. RandomAccess3. FFT
4. STREAM
5. HIMENO Benchmark (Optional)

3.Performances
4.Conclusion

7

Summary
Benchmark# Nodes

Performance

SLOC

HPL

16,384933.8 TFlops (44.5% of peak)306

RandomAccess

16,384

162.6 GUPs

250

FFT36,864

50.1 TFlops (1.1% of peak)239 + 283 + 1892

STREAM

16,384481.8 TB/s

66

HIMENO

82,944

1.3 PFlops (12.7% of peak)

137

Full compute nodes

8

HPL
Source lines of Code (SLOC) is 306, written in XMP/CBlock-Cyclic Distribution

A[N][N]

double	  A[N][N];#pragma	  xmp	  nodes	  p(P,Q)#pragma	  xmp	  template	  t(0:N-­‐1,	  0:N-­‐1)#pragma	  xmp	  distribute	  t(cyclic(NB),	  \	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  cyclic(NB))	  onto	  p#pragma	  xmp	  align	  A[i][j]	  with	  t(j,i)

12

3

412

Programmer can use BLAS for distributed array.

Panel Broadcast by using gmove directivedouble	  A_L[N][NB];#pragma	  xmp	  align	  A_L[i][*]	  with	  t(*,i)
	  	  	  	  :#pragma	  xmp	  gmove
A_L[k:len][0:NB]	  =	  A[k:len][j:NB];

A[N][N]

A_L[N][NB]

klenjNB

9RandomAccess
SLOC is 253, written in XMP/CThe XMP RandomAccess is iterated over sets of CHUNK updates oneach node
Local-view programming with XMP/C coarray syntax
u64Int	  recv[LOGPROCS][RCHUNK+1]:[∗];...
for	  (j	  =	  0;	  j	  <	  logNumProcs;	  j++)	  {	  	  recv[j][0:num]:[i_partner]	  =	  send[i][0:num];	  Deﬁne coarrayPut operation#pragma	  xmp	  sync	  memory#pragma	  xmp	  post(p(i_partner),	  0)	  	  	  :#pragma	  xmp	  wait(p(j_partner))}

A point-to-point synchronization is speciﬁed with the XMPʼs post andwait directives to realize asynchronous behavior of this algorithm
10

FFT
SLOC is 239+ 283 +1,892, written in XMP/Fortran + OpenMP
We mainly modiﬁed “pzﬀt1d.f” in ﬀte optimized for the K computerThis SLOC is 70 (the SLOC of original “pzﬀt1d.f” is 101)
The total SLOC of the modiﬁed ﬁles is 239 (70 + 169)
The total SLOC of NOT modiﬁed ﬁles is 283
The total SLOC of C interface and another kernels of the ﬀte is 1,892
CALL	  XMP_TRANSPOSE(B,A,1)...
!$XMP	  loop	  on	  tx(I)!$OMP	  parallel	  doDO	  I=1,NX	  	  DO	  J=1,NY	  	  	  	  B(J,I)=B(J,I)∗W(J,I)
	  	  END	  DOEND	  DOA() and B() areglobal arrays

A(NX,NY)
XMP̲TRANSPOSE()
B(NY,NX)

11

EP STREAM Triad
SLOC is 66, written in XMP/C + OpenMPa[], b[], and c[] are local arrays#pragma	  xmp	  nodes	  p(∗)...
for(k=0;	  k<NTIMES;	  k++)	  {#pragma	  omp	  parallel	  for	  	  for	  (j=0;	  j<size;	  j++)	  	  	  	  a[j]	  =	  b[j]	  +	  scalar∗c[j];}...
∗triadGBs	  =	  ...	  	  //	  performance	  of	  each	  node...
#pragma	  xmp	  reduction(+:triadGBs)

Parallelize the program

Gather performance resultson each node12

HIMENO Benchmark
SLOC is 137, written in XMP/Fortran
SLOC of the original is 380, written in MPI/FortranTo synchronize overlapped regions, we use XMP shadow and reﬂect
directives!$xmp	  shadow	  p(0,2:1,2:1)
	  ...
!$xmp	  reflect	  (p)	  ...
!$xmp	  loop	  (J,K)	  on	  t(*,J,K)do	  K	  =	  2,	  kmax-­‐1	  	  do	  J	  =	  2,	  jmax-­‐1	  	  	  	  do	  I	  =	  2,	  imax-­‐1	  	  	  	  	  	  S0	  =	  a(I,J,K,1)∗p(I+1,J,K)	  +	  ...	  	  	  	  	  	  	  :	  	  	  	  enddo	  	  enddo
enddo

Deﬁne overlapped region
Synchronize overlapped region
Work mapping

Only add XMP directives
into the sequential Himeno
benchmark basically.
13Results(1/3)
HPL (1 process/node +Threaded BLAS)

RandomAccess（8 processes/node）
for

K

1000$
1000

11$iz
edtim

1$

XM

P

op-1.4cc

P933.8 TFlops,
44.5% of peak
16,384 nodes(131,072 Cores)

10$

hp

10$

XM

10

Performance (GUPs)

keaoreti ca

lp

100
100$

The

Performance (TFlops)

100$

162.6 GUPs,
16,384 nodes(131,072 Cores)

0.1$

0.1
0.1$
8$
8128

1024$
2048

32768

Number of CPU Cores

0.01$
8$

1024$

131072$

Number of CPU Cores
14

Results(2/3)
FFT (1 process/node with8 threads)
K

1000$

0.1$

0.01$
8$

512$

32768$

Number of CPU Cores

10$

PP
XM
50.1 TFlops,
1.1% of peak,
36,864 nodes(294,912 Cores)

100$

XM

cc
-1

1$

Performance (TB/s)

fordize.4

op

tim

10$

hp

Performance (TFlops)

100$

STREAM (1 process/node with8 threads)

1$481.8 TB/s,
16,384 nodes(131,072 Cores)

0.1$
8$

512$

32768$

Number of CPU Cores
15

Results(3/3)
HIMENO Benchmark(1 process/node with 8 threads)
1.3 PFlops, 12.7% of peak
82,944 nodes(663,552 Cores),nc

The performance of the XMP HIMENO Benchmark
is 20% better than that of the original one.
- Threaded pack/unpack operations- Persistent communication

IMENO

Be

10$

hm

XM
P

ark

100$

ina

lH

1$
0.1$

Orig

Performance (TFlops)

1000$

0.01$
8$

1024$

131072$

Number of CPU Cores
16

Conclusion
Benchmark# Nodes

Performance

SLOCHPL

16,384933.8 TFlops (44.5% of peak)306

RandomAccess

16,384

162.6 GUPs

250

FFT36,864

50.1 TFlops (1.1% of peak)239 + 283 + 1892

STREAM

16,384481.8 TB/s

66

HIMENO

82,944

1.3 PFlops (12.7% of peak)

137

Masahiro Nakao, et al. "Productivity and Performance of the HPC Challenge
Benchmarks with the XcalableMP PGAS language", 7th International
Conference on PGAS Programming Models, October, 2013

For more information, please visit#1437 RIKEN AICS (Advanced Institute for Computational Science)#2519 Center for Computational Sciences, University of Tsukuba
17