University of Tsukuba | Center for Computational Sciences

Supercomputer at CCS: Cygnus
Multi-Hybrid Accelerated Computing Platform
Network switch
SINGLE(100Gbps x2)
NODE(without FPGA)• Combining goodness of different type of accelerators: GPU + FPGA• GPU is still an essential accelerator for simple and large degree ofparallelism to provide ~10 TFLOPS peak performance• FPGA is a new type of accelerator for application-specific hardware with
programmability and speeded up based on pipelining of calculation• FPGA is good for external communication between them with advanced
high speed interconnection up to 100Gbps x4 chan.
• Our new supercomputer “Cygnus”• Operation started in May 2019
• 2x Intel Xeon CPUs, 4x NVIDIA V100 GPUs, 2x Intel
Stratix10 FPGAs• Deneb: 48 CPU+GPU nodes• Albireo: 32 CPU+GPU+FPGA nodes
with 2D-torus dedicated network for FPGAs(100Gbpsx4)

CPU

HCA

CPU

HCA

HCA

PCIe network (switch)

GP
UGP
UGP
UGP
U

Deneb node (x48)
Network switch(100Gbps x2)

SINGLENODE(with FPGA)

Target FPGA:
Nallatech 520N

CPUHCACPU

HCA

HCA

FPGAFPGAFPGAFPGAFPGAFPGA

HCA

PCIe network (switch)PCIe network (switch)GPUGPUGPUFPGAGPUFPGAInter-FPGA
direct network(100Gbps x4)

Albireo node (x32)
Ordinary inter-node network (CPU, GPU) by IB EDR
With 4-ports x full bisection b/w

IB EDR Network (100Gbps x4/node)
…

FPGA

FPGA

Specification of Cygnus

Network switch(100Gbps x2)

Inter-FPGA
direct network(100Gbps x4)

Inter-FPGA direct network

HCA

PCIe network (switch)

Target GPU:NVIDIA Tesla V100

• FPGA design plan
• Router- For the dedicatednetwork, this impl. ismandatory.
- Forwarding packets
to destinations• User Logic- OpenCL kernel runs
here.
- Inter-FPGA comm.
can be controlledfrom OpenCL kernel.
• SL3- SerialLite III : IntelFPGA IP- Including transceivermodules for InterFPGA data transfer.
- Users don’t need tocare

Network switch(100Gbps x2)

FPGA

comp.
node…

comp.
node

comp.
nodecomp.
nodecomp.
node…

comp.
node…

Item

Specification

Peakperformance

2.4 PFLOPS DP(GPU: 2.2 PFLOPS, CPU: 0.2 PFLOPS, FPGA: 0.6 PFLOPS SP)⇨ enhanced by mixed precision and variable precision onFPGA# of nodes

80 (32 Albireo (GPU+FPGA) nodes, 48 Deneb (GPU-only)
nodes)

Memory192 GiB DDR4-2666/node = 256GB/s, 32GiB x 4 for
GPU/node = 3.6TB/s

CPU / nodeIntel Xeon Gold (SKL) x2 sockets

GPU / node

NVIDIA V100 x4 (PCIe)FPGA / nodeIntel Stratix10 x2 (each with 100Gbps x4 links/FPGA andx8 links/node)

Global File
SystemLustre, RAID6, 2.5 PB

Interconnect
ion Network

Mellanox InfiniBand HDR100 x4 (two cables of HDR200 /node)
4 TB/s aggregated bandwidthj

Programmin
g Language

CPU: C, C++, Fortran, OpenMP, GPU: OpenACC, CUDA
FPGA: OpenCL, Verilog HDLSystemVendorNEC(only for Albirero nodes)

Inter-FPGA direct network
Albireo nodes
64 FPGAs on Albireo nodes areDeneb nodes
connected directly as 2D-Torus
configuration without Ethernet sw. Ordinary inter-node communication channel for CPU and GPU, butthey can also request it to FPGA

[1]OpenCL-ready High Speed FPGA Networking
FPGA-based parallel comp. with OpenCL- Needs a communication system being
suitable to OpenCL and Intel FPGAs- Using of Intel FPGA SDK for OpenCL・ Channel Extension:
Transferring data between
kernels directly (low latencyand high bandwidth)
・ We can use multiple
kernel design to exploit
space parallelism in an
FPGA・ I/O Channel- connects OpenCL
with peripherals- We used this feature

Cluster System with FPGAs

Comm. w/o channels

Pipelined communication experiment

Our proposed method
Communication Integrated Reconfigurable
CompUting System (CIRCUS)
CIRCUS enables OpenCL code communicate
with other FPGAs on different nodes
Extending Intel’s channel mechanism toexternal communications
Pipeline manner: sending/receiving data
from/to compute pipeline directly
sender(__global float* restrict x, int n) {for (int i = 0; i < n; i++) {float v = x[i];write_channel_intel(simple_out, v);
sender code on FPGA1}}

Comm. w/ channels

receiver(__global float* restrict x, int n) {for (int i = 0; i < n; i++) {float v = read_channel_intel(simple_in);x[i] = v;
receiver code on FPGA2}
}

Recv.

Comp.

90.7Gbps↑
Send

CIRCUS
Backends[2]
Authentic Radiation Transfer

Table 2: Resource usage and clock frequency• ART is ray tracing based algorithm• problem space is divided
into meshes and reactions
are computed on each mesh• Memory access patterndepends on ray direction• Not suitable for SIMD architecture

mesh

1400

Registers (% )267,828 31%
344,447 40%
CPU(14C) 40%
344,512CPU(28C) 40%
344,505400

P100(x1)

size # of PE salmost equal
A performance
LM s (% )1200(16, 16, 16)(2, 2, 2) 132,283 31%
1000(32, 32, 32)(2, 2, 2) 169,882 40%
4.9x faster
800(64, 64, 64)(2, 2, 2) 169,549 40%
600 128, 128)
(128,(2, 2, 2) 169,662 40%

Performance [M mesh/s]• Problem space is divided into small blocks• e.g. (16, 16, 16) → 8 × (8, 8, 8)• PE is assigned to each of small blocks
b etter• Accelerated Radiative transfer on grids Oct-Tree(ARGOT) has been developer in Center for
Computational Sciences, University of Tsukuba• ART is one of algorithms used in ARGOT anddominant part (90% or more of computationtime) of ARGOT program

FPGA2000(16x16x16)• PEs are connected by channels each other• PE: Processing Element• BE: Boundary Element• Kernel of PEs and BEs are started automatically byautorun attribute• Lower control overhead and resource usagebecause of decreasing number of host controlledkernels(8x8x8)M 20
7777

Table 3: Performancecomparison
between(128,128,128)
FPGA , CPU and(16,16,16)(32,32,32)(64,64,64)
GPU implementations. T hemeshunit
sizeis M mesh/sec.
Size(16,16,16)(32,32,32)(64,64,64)(128,128,128)

CPU (14C)
112.4
158.9
175.0
95.4

CPU (28C)
77.2
183.4
227.2
165.0

P100
105.3
490.4
1041.41116.1FPGA
1282.81165.21111.01133.5

Reference[1] Norihisa Fujita, Ryohei Kobayashi, Yoshiki Yamaguchi, and Taisuke Boku, Parallel Processing on FPGA Combining Computation and Communication in OpenCL Programming, 2019 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), pp.479-488, May 2019[2] Norihisa Fujita, Ryohei Kobayashi, Yoshiki Yamaguchi, Yuuma Oobata, Taisuke Boku, Makito Abe, Kohji Yoshikawa, and Masayuki Umemura: Accelerating Space Radiate Transfer on FPGA using OpenCL (Accepted), International Symposium on Highly-Efficient Accelerators and Reconfigurable Technologies (HEART 2018)
Acknowledgment
This research is a part of the project titled “Development of Computing-Communication Unified Supercomputer in Next Generation” under the program of “Research and Development for Next-Generation Supercomputing Technology” by MEXT. We thank Intel University Program for providing us both of hardware and software.

per link) multiple interconnection links (up to 4 channels) on
it. Additionally, HLS such as OpenCL programming environment is provided, and there are several tyeps of research
to involve them in FPGA computing. I n [3], K obayashi, etal. show the basic feature to utilize the high speed interconnection over FPGA driven by OpenCL kernels. T herefore,although the performance of our implementation is almost

https://www.ccs.tsukuba.ac.jp/

weak pthroughrun ouration I nthan A
blocks a9.

R[1] K . M
F. KHeaandAstr[2] K . H
ComimaI EEApp