University of Tsukuba | Center for Computational Sciences

HA-PACS/TCA: Tightly Coupled Accelerators for Low-Latency Communication
Overview of Tightly Coupled Accelerators (TCA) Architecture and HA-PACS/TCA

GPU0

GPU1GPU3

CPU1

QPI

PCIe

GPU2

GPU3

G3 x8

infiniBand

G3 x8

infiniBand

InfiniBand Network

GPU2

G2x16

CPU0

G2x16

PCIe

G2x16

GPU1

CPU1

QPI

G2x16

G2x16

GPU0

G2x16

G2 x8

G2x 8

PEACH2CPU0
G2x16

G2 x8

G2x16

G2 x8

G2x 8

PEACH2PEACH2G2x 8

G2 x8

G2x 8

PEACH2G2x 8

G2x 8

TCA Interconnect

Block diagram ofcomputation node of HA-PACS/TCA
Port N
CPU&GPU side(Endpoint)

MemoryNios II(CPU)To PEACH2(Root Complex /Endpoint)

Routingfunction

To PEACH2(Endpoint)

PortW

DMAC

To PEACH2(Root Complex)

GPGPU is now widely used for accelerating scientific and engineering computing toimprove performance significantly with less power consumption.
However, I/O bandwidth bottleneck causes serious performance degradation on
GPGPU computing. Especially, latency on inter-node GPU communication
significantly increases by several memory copies. To solve this problem, TCA (TightlyCoupled Accelerators) enables direct communication among multiple GPUs overcomputation nodes using PCI Express.
PEACH2 (PCI Express Adaptive Communication Hub ver. 2) chip is developed andimplemented by FPGA (Field Programmable Gate Array) for flexible control andprototyping. PEACH2 board is also developed as an PCI Express extension board.
HA-PACS/TCA, which is an extended part of HA-PACS base cluster, was installedwith PEACH2 board in each node on Oct. 2013. HA-PACS/TCA is operated with HAPACS base cluster, and entire HA-PACS system becomes over 1.1 PFLOPS GPU cluster.
Currently we develop programming APIs for CUDA programmer, evaluate several
benchmarks and applications, such as QUDA (Lattice QCD library), Himenobenchmark (3D Poisson Solver), HPCG etc. We also have a poster in technicalsession, which shows block-stride transfer performance and Himeno benchmark
results.

PortE

Port S
Block diagram of PEACH2 Chip(Side View)(Top View)

TCA Communication Board (PCIe CEM Spec., double height)

Entire HA-PACS System Including
HA-PACS/TCA (5 racks x 2 rows)
HA-PACS/TCA Specification
Computation node

CRAY 3623G4-SM

Motherboard

SuperMicro X9DRG-QFCPUIntel Xeon E5 2680 v2(Ivy Bridge 2.8 GHz, 10core) x 2 socket

MemoryDDR3-1866MHz 4ch. 128 GB (119.4 GB/s)

Basic Performance of TCA Communication
CPU: Cpu-to-CPU neighbor communication, GPU: GPU-to-GPU neighbor communication
MV2: MVAPICH2, MV2GDR: with GDB on IB QDRx1
TCA latency is 3x shorter than MV2GDR

TCA bandwidth is better than MV2GDR
when message size is less than 512K

Peak Performance 448 GFLOPS/node
GPU

NVIDIA Tesla K20X x 4 GPU

MemoryGDDR5 2600MHz, 6 GB/GPU (250 GB/s/GPU)

Peak Performance 5.24 TFLOPS/nodeInterconnect

IB QDR x 2 rails (Mellanox Connect X-3)

TCA Interconnect

PEACH2 (FPGA: Altera Stratix IV 530GX)# of Nodes

64

Peak Performance

364 TFLOPS (CPU: 28.7 TF, GPU: 335.3 TF)

LINPACKBenchmark277 TFLOPS (Efficiency: 76%)
3.52 GFLOPS/W (3rd Nov. 2013 Green 500)

Ping-pong Latency using DMA

Ping-pong Bandwidth using DMA

HA-PACS Project is partially supported by the JST/CREST program entitled “Research and Development on Unified Environment of Accelerated Computing and Interconnection for Post-Petascale Era” in the research area of“Development of System Software Technologies for post-Peta Scale High Performance Computing.”

http://www.ccs.tsukuba.ac.jp/