University of Tsukuba | Center for Computational Sciences

Supercomputers at CCS: COMA and HA-PACS/TCA
COMA
Item

Specification

Peak performance

1.001 PFLOPS(MIC: 843.9TF, CPU: 157.2TF)

switchCPU1Intel E5-2670v2
P0Intel10 IVB E5-2670v2
CORE
DDR-3
10 IVB CORE P1DDR-III#2# of nodes

393

File system

Lustre, 1.5 PB user area
(DDN SFA12000)

Infiniband network switch

393 port FDR(Mellanox SX6536)

Total network bandwidth

2.75 TB/s

Language

Fortran90, C, C++

MPI

MVAPICH2, Intel MPI

System Management

Cray Advanced Cluster Engine,SLURM#3#1DMI2

E5-2670v2
P1 IntelCPU2Intel
E5-2670v2
10 IVBCORE
DDR-3
10IVBCORE
P0
DDR-III#1Intel Xeon Phi 7110p
SLOT 1/2

HEAD： CCS 名称／ CCS・ 筑波大ロ ゴ
Infiniband FDR
SLOT5Block diagram of computation node of COMACOMA (Cluster Of Manycore Architecture) is the 9th generation of PACS series supercomputer at CCS, of which computation
node is equipped with two set of Intel Xeon Phi co-processor (Knights Corner) as well as two sockets of Intel Xeon CPU (IvyBridge). The interconnection network topology is a Fat-Tree with full bisection bandwidth supported by Mellanox
InfiniBand/FDR. To avoid communication bottleneck between multiple PCIe devices over Intel QPI, all these devices areconnected to single socket of Xeon CPU.
Either of “offloading”, “native” and “symmetric” modes of Xeon Phi co-processor is available for users. Especially, the codedevelopment in the native mode is recommended to users to shift them to our new platform, Oakforest-PACS undermanagement of JCAHPC as the largest Xeon Phi (Knights Landing) cluster in Japan with 25 PFLOPS of peak performance. Several
computational science codes developed and tuned on COMA are ported to Oakforest-PACS with several times fasterperformance on each node.

HA-PACS/TCA: testbed for Tightly Coupled Accelerators concept
TCA Interconnect

GPU

NVIDIA Tesla K20X x 4 GPU

MemoryGDDR5 2600MHz, 6 GB/GPU (250 GB/s/GPU)PEACH2CPU0

G2 x8

G2x16

GPU2GPU3

CPU1

QPI

GPU0

GPU1PCIe

GPU2GPU3G3 x8infiniBand

G3 x8infini
Band

InfiniBand Network

G2x16

G2x16
GPU1

G2x16

Peak Performance 448 GFLOPS/node

G2 x8GPU0PCIeG2x16

DDR3-1866MHz 4ch. 128 GB (119.4 GB/s)

G2x 8

MemoryPEACH2CPU1QPI

G2x16Intel Xeon E5 2680 v2(Ivy Bridge 2.8 GHz, 10core) x 2 socket

CPU0
G2x16

CPUG2 x8

G2x16

SuperMicro X9DRG-QF

PEACH2G2x 8Motherboard

G2x 8

Computation node CRAY 3623G4-SM

G2 x8

G2x 8

PEACH2

G2x 8

G2x 8

HA-PACS/TCA Specification

Block diagram ofcomputation node of HA-PACS/TCA

Peak Performance 5.24 TFLOPS/nodeInterconnect

IB QDR x 2 rails (Mellanox Connect X-3)

TCA Interconnect

PEACH2 (FPGA: Altera Stratix IV 530GX)# of Nodes

64

Peak Performance

364 TFLOPS (CPU: 28.7 TF, GPU: 335.3 TF)

LINPACKBenchmark

277 TFLOPS (Efficiency: 76%)
3.52 GFLOPS/W (3rd Nov. 2013 Green 500)

(Side View)(Top View)

TCA Communication Board (PCIe CEM Spec., double height)

HA-PACS (Highly Accelerated Parallel Advanced system for Computational Sciences) is the 8th generation of PACS series
supercomputer at CCS. It consists of Base Cluster and TCA parts. TCA (Tightly Coupled Accelerators) is a concept of advanced
accelerated computing by coupling of GPU computing and low latency communication based on FPGA. As a proof-of-concept
system of TCA, we implemented PEACH2 (PCI Express Adaptive Communication Hub ver.2) with Altera Stratix-IV FPGA. All the64 nodes of HA-PACS/TCA part consists of two Intel Xeon CPU (Ivy Bridge), four NVIDIA GPU (K20X) and a PEACH2 board. As
well as the testbed for TCA concept, HA-PACS/TCA is ranked at #3 in Green500 list of November 2013 with highperformance/power ratio.
We have developed programming APIs for CUDA programmer and evaluated several benchmarks and applications, such as
QUDA (LatticeQCDlibrary), Himeno benchmark (3D Poisson Solver), etc. Recently, we implemented GASNet for GPU, which isunder development by Lawrence Berkeley National Laboratory, on PEACH2 to exploit high performance GPU-to-GPU
communication especially for halo communication on 3D stencil computation.
http://www.ccs.tsukuba.ac.jp/