2012 HPC Challenge Class 2 Submission
XcalableMP for Productivity and Performance†Masahiro Nakao, Hitoshi Murai‡

Takenori Shimosaka,‡ Mitsuhisa Sato†‡† Center for Computational Sciences, University of Tsukuba, Japan‡ RIKEN Advanced Institute for Computational Science, Japan

Outline
1.Whatʼs XcalableMP ?
2.Implementation and performance
1. HPL2. RandomAccess3. FFT
4. HIMENO Benchmark (Optional)

3.Conclusion

HPC Challenge Class II BoF@SC12, Nov. 13th

What is XcalableMP?
XcalableMP (XMP)
Directive-based language extension of Fortran 2008 and C99
XMP/Fortran language and XMP/C language exist (same directives)
Global-view and Local-view programming models
A part of the design is based on the experiences of HPF
XMP/Fortran is upward compatible with the Fortran 2008
Coarray syntax is also available in XMP/C
XMP specification working group consists of members fromacademia, research labs, and industries
Univ. of Tsukuba, RIKEN AICS, Fujitsu, NEC, Hitachi, and so onOmni XMP compiler 0.6 and XMP speciﬁcation 1.1 are available at
http://www.xcalablemp.org
HPC Challenge Class II BoF@SC12, Nov. 13th

Benchmarks
3 HPCC benchmarks : HPL, RandomAccess, FFT
Optional benchmark : Himeno benchmark
Evaluates performance of incompressible ﬂuid analysis code
Typical stencil application
Why do we select the HIMENO benchmark ?
To demonstrate parallelization by XMP directives which
communicate and synchronize the overlapped regions
for stencil applications

HPC Challenge Class II BoF@SC12, Nov. 13th

Machines
The K computer

HA-PACSCPUSPARC64 VIIIfx 2.0GHz
8Cores, 128GFlops

Xeon E5-2670 2.6GHz x2
8Cores x2, 332.8GFlops

MemoryDDR3 SDRAM 16GB
64GB/s/Socket

DDR3 SDRAM 128GB
51.4GB/s/Socket

NetworkTorus fusion six-dimensional
mesh/torus network, 5GB/s

Infiniband QDRx2rails
Fat-tree network, 4GB/s

To measure the performance, we used maximum 8192 nodesof the K computer and maximum 64 nodes of HA-PACS
HPC Challenge Class II BoF@SC12, Nov. 13thHPL(Implementation)
Source lines of code (SLOC) is 288, written in XMP/CBlock-Cyclic Distribution

A[N][N]

double	  A[N][N];#pragma	  xmp	  nodes	  p(P,Q)#pragma	  xmp	  template	  t(0:N-­‐1,	  0:N-­‐1)#pragma	  xmp	  distribute	  t(cyclic(NB),	  \	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  	  cyclic(NB))	  onto	  p#pragma	  xmp	  align	  A[i][j]	  with	  t(j,i)

12

3

4

12

Programmer can use BLAS for distributed array.

Panel Broadcast by using gmove directive
double	  A_L[N][NB];#pragma	  xmp	  align	  A_L[i][*]	  with	  t(*,i)
	  	  	  	  :#pragma	  xmp	  gmove
A_L[k:len][0:NB]	  =	  A[k:len][j:NB];

A[N][N]

A_L[N][NB]

klenjNBHPC Challenge Class II BoF@SC12, Nov. 13thHPL(Result)
The K computer
1E+03

HA-PACS

XMP
Peak of System

1E+02

TFlops

1E+02
1E+01

1E+01

170 TFlops(32% of peak)

1E+00

1E-02

1E+00

0.09 TFlops(73% of peak)

1E-01

8128

2048

17 TFlops(79% of peak)
0.15 TFlops(90% of peak)

32768

Number of CPU Cores

1E-01

16642561024

Number of CPU CoresOn HA-PACS, the performance of the XMP implementation is goodOn the K computer, the performance is a little worse on only one nodeIn this time, we did not have enough time for tuning it ...
HPC Challenge Class II BoF@SC12, Nov. 13th

RandomAccess(Implementation)
We have implemented 2 versions, written in XMP/C
For the K computer version, SLOC is 253
For general cluster version, SLOC is 278
Note that there are minor diﬀerences (only local operations arechanged)
Local-view programming with XMP/C coarray syntax
recv[j][0:num]:[i_partner]	  =	  send[i][0:num];	  //	  Put	  Operation#pragma	  xmp	  sync	  memory#pragma	  xmp	  post(p(i_partner),	  0)	  	  	  :#pragma	  xmp	  wait(p(j_partner))

A point-to-point synchronization is speciﬁed with the XMPʼs post andwait directives to realize asynchronous behavior of this algorithm

HPC Challenge Class II BoF@SC12, Nov. 13th

RandomAccess(Result)
The K computer

HA-PACS
1E+01

1E+02

XMPFujitsu

GUP/s

1E+01

104 GUP/s(65536 Cores)

1E+00

1E+00

2.4 GUP/s(1024 Cores)
1E-01

1E-01
1E-02

1E-02
8645124096 32768

Number of CPU Cores

141664256 1024

Number of CPU Cores

For comparison, we also used the custom RandomAccess whosesome local functions of hpcc-1.4 were optimized by Fujitsu.
The XMP performance and scalability are good !!
HPC Challenge Class II BoF@SC12, Nov. 13th

FFT(Implementation)
SLOC is 112+1522, written in XMP/Fortran + OpenMP + MPI/C
We rewrote only “pzﬀt1d.f” in ﬀte-5.0 which is a main kernelThis SLOC is 112 (the SLOC of original “pzﬀt1d.f” is 186)
Our implementation uses C interface and another kernels of theoriginal FFT in hpcc-1.4 (these SLOC is 1522)
This implementation is a good demonstration how to mix a XMP programwith a MPI program (MPI program calls subroutine written in XMP)
A(NX,NY)

Six-step FFT algorithm!$XMP	  distribute	  tx(block)	  onto	  p!$XMP	  distribute	  ty(block)	  onto	  p!$XMP	  align	  A(*,i)	  with	  ty(i)!$XMP	  align	  A_WORK(i,*)	  with	  tx(i)	  	  	  :!$XMP	  gmove
A_WORK(1:NX,1:NY)	  =	  A(1:NX,1:NY)	  !	  all-­‐to-­‐all

1

12

2!$XMP	  gmove

A_WORK(NX,NY)

12

12

HPC Challenge Class II BoF@SC12, Nov. 13th

FFT(Result)
The K computer

HA-PACS
1E+03

1E+04

XMP
hpcc-1.4

GFlops

1E+03
1E+02

2930 GFlops(32768 Cores)

1E+01
1E+00

1E+02

1E+00
1E-01

8128204832768

Number of CPU Cores

219 GFlops(1024 Cores)

1E+01

141664

256 1024

Number of CPU CoresThe XMP performance and scalability are good !!

HPC Challenge Class II BoF@SC12, Nov. 13th

HIMENO(Implementation)
SLOC is 115, written in XMP/Fortran
SLOC of the original is 612, written in MPI/FortranTo synchronize overlapped region, we use XMP shadow and reﬂect
directives!$xmp	  shadow	  p(0,1,1)	  	  :!$xmp	  reflect	  (p)!$xmp	  loop	  (J,K)	  on	  t(*,J,K)	  reduction	  (+:GOSA)
do	  K	  =	  2,	  kmax-­‐1	  	  do	  J	  =	  2,	  jmax-­‐1	  	  	  	  do	  I	  =	  2,	  imax-­‐11	  	  	  	  	  	  SS	  =	  (S0*a(I,J,K,4)-­‐p(I,J,K))*bnd(I,J,K)	  	  	  	  	  	  GOSA	  =	  GOSA	  +	  SS	  *	  SS	  	  	  	  	  	  	  :	  	  	  	  enddo	  	  enddo
enddo

Deﬁne overlapped region
Sync. overlapped region
Work mapping

Note:
Only add XMP directives
into the sequential Himeno
benchmark basically.

HPC Challenge Class II BoF@SC12, Nov. 13thHIMENO(Result)
The K computer

HA-PACS
1E+04

1E+05

XMP
Original

GFlops

1E+04

1E+03

1E+03

1571 GFlops(1024 Cores)

1E+025773 GFlops(8192 Cores) 1E+01

1E+02
1E+01
1E+00

1E+00
832

128

512 2048 8192

1

Number of CPU Cores

41664

256 1024

Number of CPU Cores

On the HA-PACS, the XMP performance and scalability are good !!
On the K computer, we are tuning it now ...

HPC Challenge Class II BoF@SC12, Nov. 13th

Summary(SLOC)
8800
XMP

Reference

Sorry that I made ​a mistakecounting these SLOC insubmission report

288
HPL

253 938(278)112 186

RandomAccess

FFT115612

HIMENO(only kernel)

XMP/C

XMP/C

XMP/Fortran

XMP/Fortran

HPC Challenge Class II BoF@SC12, Nov. 13th

Summary(Performance)
The K computer
BenchmarkMAX Cores

Performance

HPL

32,768170 TFlops

RandomAccess

65,536

104 GUP/s

FFT

32,7682930 GFlops

HIMENO

8,192

5773 GFlops

These results
are better thansubmission

HA-PACS
BenchmarkMAX Cores

Performance

HPL

1,024

17 TFlops

RandomAccess

1,024

2.4 GUP/s

FFT1,024

219 GFlops

HIMENO

1,024

1517 GFlops
HPC Challenge Class II BoF@SC12, Nov. 13th

Best performance and SLOC
BenchmarkMAX Cores

Performance

SLOCHPL

32,768170 TFlops

288

RandomAccess

65,536

104 GUP/s

258(278)

FFT

32,7682930 GFlops

112 + 1522

HIMENO

8,192

5773 GFlops

115

Thank you for your attention !!
For more information, please visit#3613 Center for Computational Sciences, University of Tsukuba#2247 RIKEN AICS (Advanced Institute for Computational Science)
HPC Challenge Class II BoF@SC12, Nov. 13th