University	  of	  Tsukuba	  	  | Center	  for	  Computational	  Sciences

Supercomputers	  at	  CCS:	  HA-­‐PACS/TCA	  and	  COMA
HA-­‐PACS/TCA:	  testbed	  for	  Tightly	  Coupled	  Accelerators	  concept
TCA	  Interconnect

GPU

NVIDIA	  Tesla	  K20X	  x	  4	  GPU

MemoryGDDR5	  2600MHz,	  6	  GB/GPU	  (250	  GB/s/GPU)

G2x16

GPU3

CPU1

QPI

GPU0

GPU1

PCIe

GPU2

GPU3

G3	  x8

infini
BandG3	  x8infini
Band

InfiniBand	  Network

G2x16

G2x16

CPU0

G2	  x8

GPU2

G2x16

PEACH2GPU1

PCIe

G2x16

Peak	  Performance 448	  GFLOPS/nodeG2	  x8

GPU0

G2x16

DDR3-­‐1866MHz	  4ch.	  128	  GB	  (119.4	  GB/s)

G2x	  8Memory

PEACH2CPU1

QPI

G2x16Intel	  Xeon	  E5	  2680	  v2(Ivy	  Bridge	  2.8	  GHz,	  10core)	  x	  2	  socket

CPU0

G2	  x8

G2x16

CPU

PEACH2G2x	  8SuperMicro X9DRG-­‐QF

G2x	  8Motherboard

G2	  x8

G2x	  8Computation node CRAY	  3623G4-­‐SM

PEACH2G2x	  8G2x	  8

HA-­‐PACS/TCA	  Specification

Block	  diagram	  of	  computation	  node	  of	  HA-­‐PACS/TCA

Peak	  Performance 5.24	  TFLOPS/nodeInterconnect

IB	  QDR	  x	  2	  rails	  (Mellanox Connect	  X-­‐3)TCA	  Interconnect

PEACH2 (FPGA:	  Altera	  Stratix IV	  530GX)#	  of	  Nodes

64

Peak	  Performance364	  TFLOPS	  (CPU:	  28.7	  TF,	  GPU:	  335.3	  TF)

LINPACKBenchmark277	  TFLOPS	  (Efficiency:	  76%)
3.52 GFLOPS/W	  (3rd Nov.	  2013	  Green	  500)

(Side View)(Top View)

TCA	  Communication	  Board	  (PCIe CEM	  Spec.,	  double	  height)

HA-­‐PACS (Highly Accelerated Parallel Advanced system for Computational Sciences) is the 8th generation of PACS series
supercomputer at CCS. It consists of Base Cluster and TCA parts. TCA (Tightly Coupled Accelerators) is a concept of advanced
accelerated computing by coupling of GPU computing and low latency communication based on FPGA. As a proof-­‐of-­‐concept
system of TCA, we implemented PEACH2 (PCI Express Adaptive Communication Hub ver.2) with Altera Stratix-­‐IV FPGA. All the64 nodes of HA-­‐PACS/TCA part consists of two Intel Xeon CPU (Ivy Bridge), four NVIDIA GPU (K20X) and a PEACH2 board. As
well as the testbed for TCA concept, HA-­‐PACS/TCA is ranked at #3 in Green500 list of November 2013 with highperformance/power ratio.
We have developed programming APIs for CUDA programmer and evaluated several benchmarks and applications, such as
QUDA (LatticeQCDlibrary), Himeno benchmark (3D Poisson Solver), etc. Recently, we implemented GASNet for GPU, which isunder development by Lawrence Berkeley National Laboratory, on PEACH2 to exploit high performance GPU-­‐to-­‐GPU
communication especially for halo communication on 3D stencil computation.

COMA:	  testbed	  for	  Tightly	  Coupled	  Accelerators	  concept
Item

Specification

Peak performance

1.001 PFLOPS(MIC: 843.9TF,	  CPU:	  157.2TF)# of	  nodes

393

File	  system

Lustre, 1.5	  PB	  user	  area
(DDN	  SFA12000)

Infiniband network	  switch

393	  port	  FDR
(Mellanox SX6536)

Total	  network	  bandwidth

2.75	  TB/s

Language

Fortra90,	  C,	  C++MPIMVAPICH2,	  Intel	  MPI

System Management

Cray Advanced	  Cluster	  Engine,	  SLURM

switchCPU1Intel	  E5-­‐2670v2
P0Intel	  EORE
5-­‐2670v210	  IVB	  C
DDR-­‐3	  10	  IVB	  CORE P1
DDR-­‐III

#2 #3

#1

DMI2

Intel	  CPU2
E5-­‐2670v2
Intel	  E5-­‐2670v210	  IVB	  CORE
DDR-­‐3P0 10	  IVB	  CORE
DDR-­‐III

P1#1Intel	  Xeon Phi	  7110p
SLOT 1/2

Infiniband FDRSLOT5Block	  diagram	  of	  computation	  node	  of	  COMACOMA (Cluster Of Manycore Architecture) is the 9th generation of PACS series supercomputer at CCS, of which computation
node is equipped with two set of Intel Xeon Phi co-­‐processor (Knights Corner) as well as two sockets of Intel Xeon CPU (IvyBridge). The interconnection network topology is a Fat-­‐Tree with full bisection bandwidth supported by Mellanox
InfiniBand/FDR. To avoid communication bottleneck between multiple PCIe devices over Intel QPI, all these devices areconnected to single socket of Xeon CPU.
Either of “offloading”, “native” and “symmetric” modes of Xeon Phi co-­‐processor is available for users. Especially, the codedevelopment in the native mode is recommended to users to shift them to our new platform, Oakforest-­‐PACS undermanagement of JCAHPC as the largest Xeon Phi (Knights Landing) cluster in Japan with 25 PFLOPS of peak performance. Several
computational science codes developed and tuned on COMA are ported to Oakforest-­‐PACS with several times faster
performance on each node.
http://www.ccs.tsukuba.ac.jp/