HPC Challenge Award BOF@SC09

HPC Challenge Awards Class 2

HPC Challenge Benchmarksin XcalableMP
Jinpil Lee, Mitsuhisa Sato
University of Tsukuba, HPCS lab

HPC Challenge Award BOF@SC09

What is XcalableMP?
 for distributed memory systems (PC clusters)
 directive-based language extension
 based on C and Fortran 95
 directives and other language extensions

 not new, but practical “performance-aware” parallel programming no automatic parallelization
 all actions (comm. and work mapping) are taken bydirective given by programmers

The specification has been designed by XcalableMPSpecification Working Group which consists of members
from academia, research labs and industries.

visit www.xcalablemp.org for more information

HPC Challenge Award BOF@SC09Language Overview
 execution model SPMD(Single Program Multiple Data): MPI-like starting with single thread per process explicit parallelism
 no automatic communication, virtual shared memory

 Two different programming models in one language
 global view model incremental parallelization with directives (OpenMP-like) many concepts from HPF (High Performance Fortran)

 local view model supports PGAS features (co-array from Co-Array Fortran)

HPC Challenge Award BOF@SC09

Global View Programming Model incremental parallelization with directives

 adding directives to serial code programmer describes data distribution, work mapping

inter-node comm supports typical techniques for data/task parallelization

double array[YMAX][XMAX];#pragma xmp nodes p(*)#pragma xmp template t(XMAX, YMAX)#pragma xmp distribute t(*, BLOCK)
main() {int i, j, res = 0;
#pragma xmp loop on t(*,i) reduction (+:res)for (i = 0; i < YMAX; i++)for (j = 0; j < XMAX; j++) {array[i][j] = func(i,j);
res += array[i][j];}}// declare node group (communicator)// declare template (range of array index)// distribute template// work sharing and reduction

HPC Challenge Award BOF@SC09

Local View Programming Model PGAS model taken from Co-Array Fortran(extended to C)
 one-sided communication using language extension
 high interoperability with MPI
double array[YMAX/PROCS][XMAX];#pragma xmp coarray array// data distribution (manual)// declare array as a co-arraymain() {int i, j, res = 0, res_local = 0;
for (i = 0; i < YMAX/PROCS; i++)for (j = 0; j < XMAX; j++) {array[i][j] = func(i,j);
res_local += array[i][j];}for (i = 1; i <= PROCS; i++)res += res_local:[i];}// loop parallalization (manual)// access res_local on node(i)HPC Challenge Award BOF@SC09

Submission
 XMP/C: a prototype compiler is implemented
 supports basic functions for data parallelism

 In this submission, we focus on programmabilityof XcalableMP STREAM, RandomAccess, HPL, FFT are parallelized by

XMP

T2K OpenSupercomputer – Tsukuba System (2to32nodes)
CPUMEM

AMD Opteron Quad-core 8000series
2.3Ghzx 4sockets (16 cores)
32GB

NETWORKInfiniBand (x4 rails)

MPI lib

MVAPICH2 - 1.2

HPC Challenge Award BOF@SC09

Benchmark1: STREAM
 global view programming with directives
 very straightforward to parallelize by loop

directive

double a[SIZE] , b[SIZE] , c[SIZE];#pragma xmp nodes p(*)#pragma xmp template t(0:SIZE−1)#pragma xmp distribute t(block) onto p#pragma xmp align [j] with t(j) :: a, b, c...
# pragma xmp loop on t(j)for (j = 0; j < SIZE; j++) a[j] = b[j] + scalar*c[j];...
#pragma xmp reduction(+:triadGBs)

HPC Challenge Award BOF@SC09

Performance of STREAM
 Lines Of Code: 98

800
Performance (GB/s)

700
600
500
400
300
200
10002

46810 1214 1618 20

22 24

Number of Nodes (15cores per node)

26 28

30 32

HPC Challenge Award BOF@SC09

Benchmark2: RandomAccess local view programming with co-array#define SIZE TABLE_SIZE/PROCS
u64Int Table[SIZE] ;#pragma xmp nodes p(PROCS)#pragma xmp coarray Table [PROCS]...
for (i = 0; i < SIZE; i++) Table[i] = b + i ;...
for (i = 0; i < NUPDATE; i++) {temp = (temp << 1) ˆ ((s64Int)temp < 0 ? POLY : 0);
Table[temp%SIZE]:[(temp%TABLE_SIZE)/SIZE] ˆ= temp;}#pragma xmp barrier

HPC Challenge Award BOF@SC09

Performance of RandomAccess Lines Of Code: 77
 complied into MPI2 one-sided functions

0.000045
0.00004
0.000035
GUP/s

0.000030.000025
0.000020.000015
0.00001
0.000005
02468

10 12 14 16 18 20 22 24 26 28 30 32
Number of Nodes

HPC Challenge Award BOF@SC09

Gmove DirectiveData transfer (copy) in global view

Very powerful directive to describe communication
 This operation is “collective”, that is, executed by all node, andtranslated into two-sided comm.


Array section can be used in C

#pragma xmp nodes p(4)#pragma xmp template t(N)#pragma xmp distribute t(block) onto p#pragma xmp align a[*][i] with t(i)...
#pragma xmp gmove
L[0:N-1] = a[0][0:N-1]; // broadcast
L[N]

a[N][N]0 1 2 3

HPC Challenge Award BOF@SC09

Benchmark3: HPL
 parallelized in global view matrix/vectors are distributed in cyclic manner in one

dimension.
 using gmove to exchange columns for pivot exchange
dgefa function:#pragma xmp gmove
pvt_v[k:n-1] = a[k:n-1][l];if (l != k) {#pragma xmp gmove
a[k:n-1][l] = a[k:n-1][k];#pragma xmp gmove
a[k:n-1][k] = pvt_v[k:n-1];}

HPC Challenge Award BOF@SC09

Performance of HPL
 Lines Of Code: 243

Performance (Gflop/s)

1816141210864202468101214161820

Number of Nodes

222426283032

HPC Challenge Award BOF@SC09

Benchmark4: FFT parallelized in global view
 Using six-step FFT algorithm Matrix transpose is a key operation.

 matrix transpose using gmove#pragma xmp align a_work[*][i] with t1(i)#pragma xmp align a[i][*] with t2(i)#pragma xmp align b[i][*] with t1(i)...
#pragma xmp gmove
a_work[:][:] = a[:][:];// all-to-all#pragma xmp loop on t1(i)for(i = 0; i < N1; i++)for(j = 0; j < N2; j++)c_assgn(b[i][j], a_work[j][i]);HPC Challenge Award BOF@SC09

Performance of FFT
 Lines Of Code: 217

Performance (Gflop/s)

2.5
2
1.5
10.5
0
2

4

681012 141618 20 22 24

Number of Nodes

2628 3032

HPC Challenge Award BOF@SC09

Future Works Full implementation of gmove directive, task

parallelism ...
 More improvement of the performance, and algorithm.
 One-sided communication
 2-D blocking in HPL hybrid programming for SMP clusters:

XMP with OpenMPfor more information. . .

visit #639

Center for Computational Sciences,
University of Tsukuba#2897 T2K Open Supercomputer Alliance