PACS-X
New Frontier of Accelerated HPC

Taisuke Boku
Deputy Director / HPC Division Leader
Center for Computational Sciences
University of Tsukuba

1

2017/12/07

CCS-EPCC-WS2017@Tsukuba
Center for Computational Sciences, Univ. of Tsukuba

Outline
FPGA for HPC as large scale parallel system
n AiS (Accelerator in Switch) concept
n FPGA for high performance interconnection andcomputationn PACS-X Project and PPXn OpenCL-enabled computation/communication on FPGAn Application Example: Astrophysics
n Summaryn2

CCS-EPCC-WS2017@Tsukuba
2017/12/07

Center for Computational Sciences, Univ. of Tsukuba

Accelerators in HPC
nTraditionally...
Cell Broadband Engine, ClearSpeed, GRAPE. ...
n then GPU (most popular)
and MATRIX-2000 J
nn

Is GPU perfect ?
good for many applications (replacing vector machines)
n depending on very wide and regular computation
nlarge scale SIMD (STMD) mechanism in a chipn high bandwidth memory (GDR5, HBM) and local memorynn

bad fornot enough parallelism
n not regular computation (warp spliting)n frequent inter-node communication (kernel switch, go back to CPU)n3

CCS-EPCC-WS2017@Tsukuba
2017/12/07

Center for Computational Sciences, Univ. of TsukubaFPGA in HPCn

Goodness of recent FPGA for HPC
nnnnnnTrue codesigning with applications (essential)
Programmability improvement: OpenCL, other high level languages
High performance interconnect: 40Gb~100Gb
Precision control is possible
Relatively low power

Problemsnnn

4

Programmability: OpenCL is not enough, not efficient
Low standard FLOPS: still cannot catch up to GPU-> “never try what GPU works well on”
Memory bandwidth: 2-gen older than high end CPU/GPU-> be improved by HBM (Stratix10)

CCS-EPCC-WS2017@Tsukuba
2017/12/07

Center for Computational Sciences, Univ. of Tsukuba

Simple pros/consperformance(FLOPS)

externalcommunication(sec, B/s)

programmingcostCPU△○◎GPU◎△○FPGA○◎×➝△?

How to compensate with each othertoward large degree of strong scaling ?

5

CCS-EPCC-WS2017@Tsukuba
2017/12/07

Center for Computational Sciences, Univ. of Tsukuba

AiSnAiS: Accelerator in Swtich
Using FPGA not only for computation
offloading but also for communicationn Combining computation offloading andcommunication among FPGAs for ultralow latency on FPGA computingn Especially effective on communicationrelated small/medium computation(such as collective communication)n Covering GPU non-suited computation byFPGAn OpenCL-enable programming for
application users
n6GPUCPU

PCIeFPGAcomp.
comm.

high speed interconnect

CCS-EPCC-WS2017@Tsukuba
2017/12/07

Center for Computational Sciences, Univ. of Tsukuba

AiS computation model
invoke GPU/FPGA kernsls

CPU

PCIe

GPU

data transfer via PCIe
(invoked from FPGA)CPUFPGAcomp.

PCIe

comm.

collective or specializedcomp.+comm.

GPU
FPGAcomp.
comm.

> QSFP+ interconnect

Ethernet Switch

7

CCS-EPCC-WS2017@Tsukuba
2017/12/07

Center for Computational Sciences, Univ. of Tsukuba

How fast on communication ? (FPGA-FPGA link)parallel link bundup to 3 channel

nnnn8

Xilinx XC7VX1140T(Virtex7) with 100Gbps optical interconnect
up to 96% of theoretical peak
good scalability up to 3 channels aggregation
Intel Stratix10 will have up to 4 channels of 100Gbps link

CCS-EPCC-WS2017@Tsukuba
2017/12/07

Center for Computational Sciences, Univ. of Tsukuba

PACS-X (ten) Project at CCS, U. Tsukuban

PACS (Parallel Advanced system for Computational Sciences)nn

a series of co-design base parallel system development both on systemand application at U. Tsukuba (1978~)recent systems focus on accelerators
PACS-VIII: HA-PACS (GPU cluster, Fermi+Kepler, PEACH2, 1.1PFLOPS)n PACS-IX: COMA (MIC cluster, KNC, 1PFLOPS)
nn

Next generation of TCA implementation
nnnn

PEACH2 with PCIe is old and with several limitation
new generation of GPU and FPGA with high speed interconnection
more tightly co-designing with applications
system deployment starts from 2018 (?)

PPX: Pre-PACS-X
9CCS-EPCC-WS2017@Tsukuba
2017/12/07

Center for Computational Sciences, Univ. of Tsukuba

PPX: testbed under AiS concept (x6~12 nodes)
1.6TB NVMe

Xeon Broadwell
QPI

HCA:Mellanox IB/EDR

100G IB/EDR
GPU: coarse-grain
offloading
NVIDIA P100 x 2

Xeon Broadwell

40Gb Ether x 2-> upgrade to 100G x 2
10

FPGA:
fine-grain
partial offloading+
high-speed interconnect
Altera Arria10(Bitware A10PL4)

CCS-EPCC-WS2017@Tsukuba
2017/12/07

Center for Computational Sciences, Univ. of Tsukuba

PPX (Pre-PACS-X) mini-cluster system
InfiniBand/EDR Switch
InfiniBand/EDR (100Gbps)
x12 nodes

computenodeCPU: BDW x 2
GPU: P100 x 2
FPGA: Intel or Xilinx

....
FPGA:
6 with Intel
6 with Xilinx

Ethernet (40Gbps x 2)

100G Ethernet Switch

GbE switch

login node

11CCS-EPCC-WS2017@Tsukuba
2017/12/07

Center for Computational Sciences, Univ. of Tsukuba

OpenCL-enabled high speed network
nOpenCL environment is availablennn

But, current FPGA board is not ready for OpenCL oninterconnect accessnn

BSP (Board Supporting Package) is not complete for interconnect➝ we developed for OpenCL accessOur goalnnn12ex) Intel FPGA SDK for OpenCL
basic computation can be written in OpenCL without Verilog HDL

enabling OpenCL description by users including inter-FPGAcommunication
providing basic set of HPC applications such as collective
communication, basic linear libraryproviding 40G~100G Ethernet access with external switches for large
scale systems

CCS-EPCC-WS2017@Tsukuba
2017/12/07

Center for Computational Sciences, Univ. of Tsukuba

BSP (Board Support Package)• description specifying FPGA chip and board peripherals
configuration and access/control method- independent for each board with FPGA
- a sort of virtualization to enable same kernel development on FPGA

• minimum interface is provided by board vendors- we need optical interconnection access method in BSPadditionally developed

FPGA board

optical interconnect port

FPGA

QSFP+Controller

OpenCLkernelgenerated fromKernel code

DRAM
DDR4Controller
PCIeController

Driver

HostProgramCPU
generatedfrom BSP

13

CCS-EPCC-WS2017@Tsukuba
2017/12/07

Center for Computational Sciences, Univ. of Tsukuba

Our test bed (BittWare A10PL4 with Intel Arria10 FPGA)I/O channel
specification

FPGA

Controller
as wrapper

Ethernet IP(by Intel)

QSFP+ port
carribration

BSP

OpenCL
KernelEthernet IP
Controller

EthernetIP

QSFP+

Ethernet IP
Controller

EthernetIP

QSFP+DRAM

DDR4Controller

DRAM

PCIeController

CPU14

Host Program

Driver

CCS-EPCC-WS2017@Tsukuba
2017/12/07

Center for Computational Sciences, Univ. of Tsukuba

Ethernet IP Controller
user specified
send/recv data from OpenCL

OpenCL Kernel
recv data

MAC addres+ etc.

send datann

FIFO

joint module for OpenCL andEthernet IP
creating Ethernet framen

FIFO

INSERTn

ペイロード
ペイロード
ペイロード

n

ペイロード
ペイロード
ペイロード

payloadpayload

INSERT

EXTRACT
MAC addr.

EXTRACTnn

MAC addr. etc.
getting payload data

currently, re-transmission
feature is not implementedpayload

Ethernet IP Controller

Ethernet IP

QSFP+implemented by Verilog HDL
15

CCS-EPCC-WS2017@Tsukuba
2017/12/07

Center for Computational Sciences, Univ. of Tsukuba

OpenCL code example for pingpong
write̲channel̲intel (SET̲SOURCE , source̲addr);
write̲channel̲intel (SET̲DEST , (int2)(data̲size , dest̲addr));for( i = 0 ; i < data̲size ; i ++)write̲channel̲intel (SEND , send̲data[i] );senderfor( i = 0 ; i < data̲size ; i ++)receive̲data[i] = read̲channel̲intel (RECEIVE);
OpenCL Kernel
sender MAC addr.:
source̲addr

receiverEthernet IP Controller
SET̲SOURCE

INSERT

receiver MAC addr. : dest̲addr
SET̲DEST

FIFO

data pointer : send̲dataSEND

FIFO

data pointer : receive̲dataRECEIVE

send data size : data̲size

EXTRACT

Channel_ID

16

CCS-EPCC-WS2017@Tsukuba
2017/12/07

Center for Computational Sciences, Univ. of Tsukuba

Evaluation test-bedn

Pre-PACS-X (PPX)nncomp. node

HCA:Mellanox IB/EDR

CCS, U. Tsukuba
PACS-X prototype

IB/EDR : 100Gbps
CPU:XeonE5-2660 v4

GPU:
NVIDIA P100 x2

CPU:XeonE5-2660 v4

FPGA:Bittware A10PL4

QSFP+ : 40Gbpsx2

17Host OS

CentOS 7.3

Host Compiler

gcc 4.8.5

FPGACompiler

Intel FPGA SDK for OpenCL,Intel Quartus Prime Pro
Version 17.0.0 Build 289

CCS-EPCC-WS2017@Tsukuba
2017/12/07

Center for Computational Sciences, Univ. of Tsukuba

Communication paths
IB (InfiniBand) Switch
IB EDR(100Gbps)

NODE

PCIe Gen.3x16

CPU0

via-IBIB HCA

CPU0

IB HCA
NODE

QPI

CPU1

FPGA

CPU1・・・・

FPGA

PCIe Gen.3x8(56Gbps)
QSFP+(40Gbps)

via-Ethernet
Ethernet Switch

18

CCS-EPCC-WS2017@Tsukuba
2017/12/07

Center for Computational Sciences, Univ. of Tsukuba

Communication latency
node-to-node communication2経路間における通信レイテンシの比較
latency (1 byte data)(通信データサイズ: 1 byte)
29.03
FPGA<->CPU1better

レイテンシ[us]Latency [usec]

2527.70
FPGA -> CPUCPU -> FPGAの合計

15
1001.33
IB経由

via IB

19n

CPU-FPGA interface by current
BSP is not good

CPU1->CPU1205n~1µs latency on Ethernet
CPU-FPGA comm. occupies
comm. latency via IB

0.99

viaEthernet経由
Ethernet

1.2

Latency [usec]
レイテンシ[us]

30n

break
down of Ethernet latency
Ethernet経由のレイテンシの内訳

10.8

Ethernet IPController

0.6

Ethernet IP

0.4

Ethernet
Switch0.20

via Ethernet
Ethernet経由

CCS-EPCC-WS2017@Tsukuba
2017/12/07

Center for Computational Sciences, Univ. of Tsukuba

Communication bandwidth• 40Gbps Ethernet achieves 4.97GB/s- 99.8 % of theoretical peak (w/o error handling)- small N1/2 by short latency• via-IB achieves 2.32GB/s- non-pipelined- no special feature (such as GPUDirect) on FPGA-HCA
node-to-node communication
2経路間における通信バンド幅の比較
bandwidthBandwidth [GB/s]
バンド幅[GB/s]

better

654IB経由

3Ethernet経由

21016162562564096
4K655361048576
64K
1M
データサイズ[byte]16777216
16M

268435456
256M

Data size [Byte]

20

CCS-EPCC-WS2017@Tsukuba
2017/12/07

Center for Computational Sciences, Univ. of Tsukuba

AiS application example: ARGOTn

ARGOT (Accelerated Radiative transfer on grids using Oct-Tree)n Radiative transfer simulation code developed in CCS
n Two basic computing methods for radiation transfer
n

ARGOT methodnnfrom a light sourceARTnfrom spatially spread light sources

CPU version and GPU version with MPI
n ART method occupies >90% of computation even on GPU, and weneed more speedup➝ making FPGA offloading in AiS conceptn21

CCS-EPCC-WS2017@Tsukuba
2017/12/07

Center for Computational Sciences, Univ. of Tsukuba

ART method
radiative transfer computing on spatiallyspread light sources
n ray-tracing on 3-D space with griddecomposed partitions
nnnnnnrays are in paralleldifferent input angles
no reflection nor refraction (different from 3-Dgraphics ray-tracing)
HEALPix algorithm for ray generation

large scale for parallel processingnn

22mesh size: 1003〜10003
ray angles: 768〜1000s

CCS-EPCC-WS2017@Tsukuba
2017/12/07

Center for Computational Sciences, Univ. of TsukubaPerformance (single FPGA) on ART method
DevicePerf. [M mesh/sec]

vs CPUCPU

117.49-FPGA@228.57MHz(w/o autorun)
FPGA@236.11MHz(w/ autorun)

593.11

5.05

1714.97

14.60

up to 14.6x faster than CPU, and 5.1x faster than GPU
n 93% of computation time of ARGOT is dominated by ARTmethod➝ 7.48x speedup on entire code is expectedn23

CCS-EPCC-WS2017@Tsukuba
2017/12/07

Center for Computational Sciences, Univ. of Tsukuba

Circuit resource utilization
ALMsRegisters

M20K

MLAB

MLAB size

DSPFreq.

w/oautorun

228,610(54%)

473,747(55%)

1,839(68%)

4,330

47,968bits

536(35%)228.57
MHzw/autorun

228,835(54%)

467,225(55%)

1,716(63%)

7,350

138,288bits

536(35%)236.11
MHz+225-6,255-123+3,020+90,320

0+7.54difference

largest resource use is on M20K (63%)

nnactually 53.3% (without BSP use)

DSP utilization is only 53%n We can achieve up to 2x more speed
n24

CCS-EPCC-WS2017@Tsukuba
2017/12/07

Center for Computational Sciences, Univ. of Tsukuba

Next Stepn

Precision controllingnnn

Combining Communication and Computationnnnfor ART and ARGOT, SP is too much, HP is not balanced
finding best (e, m, s) combination
e=exponent m=mantissa s=(exponent digit shift)
OpenCL computing kernels binding with OpenCL Ethernet communication
layer kernels with OpenCL Channel (by Intel SDK)
Possibility on network from Ethernet-switch to Direct Link between FPGA
(with Stratix10 or similar in Xilinx)

Combining GPU and FPGAnn25

GPU global memory access via PCIe from FPGA
➝ technology porting from TCA/PEACH2 in HA-PACS project
GPU/FPGA offloading control from CPU➝ new programming paradigm is requiredCCS-EPCC-WS2017@Tsukuba
2017/12/07

Center for Computational Sciences, Univ. of Tsukuba

High Level Programming ParadigmnXcalableACCnnnnn

under development in collaboration between CCS-Tsukuba and RIKEN-AICS
PGAS language XcalableMP is enabled to imply OpenACC for sophisticatedcoding of distributed memory parallelization with accelerator
inter-node communication among FPGA can be implemented by FPGAEthernet direct link
Data movement between GPU and FPGAOpenACC for FPGAnnnn26(plan) research collaboration with ORNL FTG
OpenACC -> OpenCL -> FPGA compilation by OpenARC project is underdevelopmentfinal goal: XcalableACC with OpenARC compiler and FPGA-Ethernet link
(Or OpenMP->OpenCL->FPGA for XcalableMP implementation)

CCS-EPCC-WS2017@Tsukuba
2017/12/07

Center for Computational Sciences, Univ. of Tsukuba

Summary
FPGA for HPC is very attractive theme for next generation ofaccelerated platformn FPGA is usable not only for computing but also forcommunication
n 360-degree system to cover highly parallel STMD computing by
GPU and flexible processing on FPGA with communication featuren OpenCL-enabled programming including communication forapplication users
n CCS, U. Tsukuba is moving forward to realize AiS concept on next
generation multi-hetero supercomputing toward PACS-X
implementationn27

CCS-EPCC-WS2017@Tsukuba
2017/12/07

Center for Computational Sciences, Univ. of Tsukuba