HPCC BoF@SC10

HPC Challenge Awards 2010 Class2XcalableMP Submission	
Jinpil Lee, Masahiro Nakao, Mitsuhisa Sato	
University of Tsukuba	

1	

HPCC BoF@SC10

Submission Overview	 XcalableMP Language and model, proposed by XMP spec WG
 Fortran + Language extension
 OpenMP-like directives, co-array syntax
 C version compiler is used for the submission

 Submission
 3 HPCC benchmarks: Linpack, FFT, RandomAccess 2 other kernals: Laplace, NPB-CG Omni XcalableMP Compiler 0.5 (alpha) isreleased
 download at www.xcalablemp.org2	HPCC BoF@SC10

Evaluation Platform	
T2K OpenSuperComputer Tsukuba System (using 1 ~ 64 nodes)	
CPU	AMD Opteron Quad-core 8800 2.3Ghz

MEMORY	

32GB	x 4 sockets (16 cores)	

NETWORK	 Infiniband DDR (4 rails)	
OS	

Linux kernel 2.6.18 x86 64	

MPI	

MVAPICH2 1.2	Cray XT5 (using 1 ~ 16 nodes)	
CPU	AMD Opteron Shanghai 2.7Ghx

MEMORY	

32GB	

x 2 sockets (8 codes)	

NETWORK	 SeaStar 3D-Torus interconnect	
OS	

CLE 3.1 pre-release (based on Linux 2.6.27.48)	

MPI	

Cray MPT(Message Passing Toolkit) 5.1 (based on MPICH2)	3	

HPCC BoF@SC10

Parallelizing Laplace	double u[XSIZE][YSIZE], uu[XSIZE][YSIZE] ;#pragma xmp nodes p(NPCOL, NPROW)#pragma xmp template t(0:XSIZE−1, 0:YSIZE−1)#pragma xmp distribute t(block, block) onto p#pragma xmp align u[j][i] with t(i, j)#pragma xmp align uu[j][i] with t(i, j)#pragma xmp shadow uu[1:1][1:1]...
Data Distribution	
for (k = 0 ; k < niter; k++) {#pragma xmp reflect uu
Data Sync	
#pragma xmp loop (x, y) on t(y, x)for (x = 1 ; x < XSIZE−1; x++)Work Sharing	for (y = 1 ; y < YSIZE−1; y++)u [x][y] = ( uu [x−1][y] + uu [x +1][y]+ uu [x][y−1] + uu [x][y + 1]) / 4.0;
4	

HPCC BoF@SC10

Performance of Laplace (T2K-Tsukuba)	
200	
180	

per Node(XMP)	

160	

per Node(MPI)	

GFlops	140	

per Soket(XMP)	

120	

per Soket(MPI)	

100	

per Core(XMP)	

80	

per Core(MPI)	

60	
40	
20	0	
1	2	4	8	16	32	

64	

128	

256	512	 1024	

Num	  of	  XMP	  Procs	
5	

HPCC BoF@SC10

Parallelizing Linpack	 1-dimensional Cyclic Distribution	 Selecting Pivot and Exchange	 needs internode communication	 vector exchange using gmove 	

kl
pvt_v

exchange	

dgefa function:#pragma xmp gmove
pvt_v[k:n-1] = a[k:n-1][l];if (l != k) {#pragma xmp gmove
a[k:n-1][l] = a[k:n-1][k];#pragma xmp gmove
a[k:n-1][k] = pvt_v[k:n-1];}
6	HPCC BoF@SC10

Performance of Linpack (T2K-Tsukuba)	
2500	
per Node(XMP)	

MFlops	  2000	

per Soket(XMP)	
per Core(XMP)	

1500	

1000	

500	

0	1	2	4	8	16	32	64	Num	  of	  XMP	  Procs	  128	256	512	 1024	
7	

HPCC BoF@SC10

Parallelizing FFT	 six-step FFT algorithm needs three matrix transposefor (i = 0; i < N1; i++)for (j = 0; j < N2; j++)b[i][j] = a[j][i]; 1-dimensional block distribution	
 transpose needs internode comm	 implemented by gmove	

a[N2][N1]	  0	  0	  1	  1	  b[N1][N2]	  local	  copy	  0	  0	  1	  1	  gmove	  	  	  a_work[:][:]	  =	  a[:][:];	  0	  1	  0	  1	  a_work[N2][N1]	  8	

HPCC BoF@SC10

Source Code of FFT	#pragma xmp align a_work[*][i] with t1(i)#pragma xmp align a[i][*] with t2(i)#pragma xmp align b[i][*] with t1(i)...
#pragma xmp gmove
a_work[:][:] = a[:][:];// collect elmts#pragma xmp loop on t1(i)for(i = 0; i < N1; i++)for(j = 0; j < N2; j++)c_assgn(b[i][j], a_work[j][i]);// local copy
#pragma xmp loop on t1(i)for(i = 0; i < N1; i++)
HPCC_fft235(b[i], work, w2, N2, ip2); // 1-dim FFT

9	

HPCC BoF@SC10

Performance of FFT (T2K-Tsukuba)	
6	per Node(XMP)	
5	per Node(MPI)	
per Soket(XMP)	

GFlops	4	per Soket(MPI)	
per Core(XMP)	

3	per Core(MPI)	
2	

1	0	
1	2	

4	

8	

16	32	64	Num	  of	  XMP	  Procs	

128	

256	

512	 1024	
10	

HPCC BoF@SC10

Parallelizing NPB-CG	#pragma xmp template t(0:N-1,0:N-1)p[], q[], and w[] are#pragma xmp distribute t(block, block) on p
distributed arrays.	
#pragma xmp align a[j][i] to t(i,j)#pragma xmp align p[i] to t(i,*)#pragma xmp align w[j] to t(*,j)….
p[i], q[i] with t(i, *)for(){
w[i] with t(*, i)	
…
#pragma xmp loop on t(*, j)for (j = 1; j <= lastrow-firstrow+1; j++) {sum = 0.0;
for (k = rowstr[j]; k < rowstr[j+1]; k++) {sum = sum + a[k]*p[colidx[k]];}w[j] = sum;
Reduction operation}#pragma xmp reduction(+:w) on p(*, :)on replicated array	#pragma xmp gmove
q[:] = w[:];
copy arrays with….
different distributions	….. Update p with q ….
}	11	

HPCC BoF@SC10

Data Movement in NPB-CG	t（i,j)	
a[][]	
iw[j] with t(*,j)	

p[i] with t(i,*)	

×	j	

reduction	
reduction(+:w) on p(*, :)gmove q[:] = w[:];	
transpose	

12	

HPCC BoF@SC10

Performance of NPB-CG (T2K-Tsukuba)	
35000	
per Node(XMP)	

30000	

Mop/s	

per Node(MPI)	
25000	

per Soket(XMP)	

20000	

per Soket(MPI)	
per Core(XMP)	

15000	
per Core(MPI)	
10000	
5000	
0	1	2	4	8	16	32	64	

Num	  of	  XMP	  Procs	

128	 256	 512	 1024	
13	

HPCC BoF@SC10

Parallelizing RandomAccess	 Local View Programming with Co-array	

14	

HPCC BoF@SC10

Performance of RandomAccess (T2K-Tsukuba)	
0.035	
0.03	per Node(XMP)	
per Node(MPI)	

GUP/s	

0.025	
0.02	
0.015	
0.01	
0.005	
0	2	4	8	Num	  of	  XMP	  Procs	16	32	
15	

HPCC BoF@SC10

Thanks for your attention	
for more information. . .

visit #3213 Center for Computational Sciences,
University of Tsukuba#1321 T2K Open Supercomputer Alliance16	HPCC BoF@SC10

Performance of Laplace (Cray XT5)	
35	
per Node(XMP)	

30	

per Node(MPI)	

GFlops	25	

per Soket(XMP)	
per Soket(MPI)	

20	per Core(XMP)	
15	per Core(MPI)	

10	5	0	1	2	4	8	16	

Num	  of	  XMP	  Procs	32	64	

128	17	

HPCC BoF@SC10

Performance of Linpack (CRAY XT5)	
1600	
1400	

per Node(XMP)	
per Soket(XMP)	

1000	

per Core(XMP)	

MFlops	

1200	

800	
600	
400	
200	
0	
1	2	4	8	16	Num	  of	  XMP	  Procs	32	64	

128	
18	

HPCC BoF@SC10

Performance of FFT (Cray XT5)	

GFlops	18	
16	

per Node(XMP)	

14	

per Node(MPI)	

12	per Soket(XMP)	

10	per Soket(MPI)	

8	per Core(XMP)	

6	per Core(MPI)	

4	2	0	
1	2	4	

8	16	Num	  of	  XMP	  Procs	32	64	

128	19	HPCC BoF@SC10

Performance of NPB-CG (Cray XT5)	

Mop/s	

20000	
18000	

per Node(XMP)	

16000	

per Node(MPI)	

14000	

per Soket(XMP)	

12000	

per Soket(MPI)	

10000	

per Core(XMP)	

8000	

per Core(MPI)	

6000	
4000	
2000	
0	1	

2	3	

4	5	

Num	  of	  XMP	  Procs	

6	7	8	20	

HPCC BoF@SC10

Performance of RandomAccess (Cray XT5)	
0.045	

GUP/s	

0.04	0.035	

per Node(XMP)	

0.03	

per Node(MPI)	

0.025	
0.02	
0.015	
0.01	
0.005	
0	2	

4	

8	16	

Num	  of	  XMP	  Procs	
21