Accelerator-in-Switch: a novel cooperation framework forFPGAs and GPUs
− Building a Virtual Heterogeneous Computing System −2018.12Hideharu Amano
Keio University
This work is based on results obtained from a project commissioned by the New Energy andIndustrial Technology Development Organization (NEDO)
Special Thanks to T.Kudoh, R.Takano, T.Ikegami, M.Koibuchi, A.B. Ahmed, K.Hironaka,K.Musha, K.Azegami, K.Iizuka, Y.Yamauchi, M.Yamakura and other members of the project.

FPGA computing in cloud• High Performance FPGAs are available:• High computational power: Intel Stratix-10• With a large memory: Xilinx UltraScale+ with UltraRAM• But they are expensive for most users to keep themselves.

• Programming environment is improved:• Open-CL is widespread for computational usage.
• Vivado-HLS is popularly used for general usage.

→ FPGAs in cloud:More flexible and power efficient than using GPU.
• FPGA in the Cloud: Booting Virtualized Hardware Accelerators withOpenStack [FCCM2014]• Microsoft Catapult [ISCA2014 ][HEART2017]• FPGA Supervessel Cloud by IBM[ICFPT2016]• Amazon EC2 F1 Instance [https://aws.amazon.com/ec2/instance-types/f1/]

Conventional FPGA-in-Cloud…

FPGA…
PCIePCIePCIeFPGA

HostPC

HostPC

HostPCFPGAHigh-Performance FPGAs are attached into each Host.
• The total cost becomes large.
• High performance FPGAs are still expensive.
• The size of FPGA is limited.
• Multiple FPGAs cannot be used together.

Our Proposal : Virtual Large FPGAHostPC
PCIeFPGA

FPGA

FPGA

FPGA

FPGA

FPGA

FPGA

A lot of cost-efficient middle-scale FPGAs aretightly connected.
They can be treated as if they were a single FPGA inHLS description level.
Higher performance per cost than conventional
FPGA in cloud.
Practically infinite resource is used.
Separated into a number of virtual FPGAs andshared by the multiple users.

Flow-in-Cloud (FiC) is the first prototype.

Microsoftʼs Catapult V1/V2[Putnum：ISCA-2014][Caulfield：Micro-2016]

Rank computation for Web search on Bing.
Task Level Macro-Pipelining (MISD)
FE: Feature Extraction
FFE: Free Form Expression: Synthesis of feature values
MLS: Machine Learning Scoring
FE
FPGA

FFE0

FFE1

Compress

MLS0

MLS1

MLS2

FPGA

FPGA

FPGAFPGAFPGAFPGAFPGACPUCPUCPUCPUCPUCPUCPUCPUFPGA: Intel Stratix V
2-Dimensional Mesh is formed (8x6) for 1 cluster.

10Gbps network is upgraded to 40Gbps network in V2

https://euroexa.eu/
Recent FPGA supercomputers(For example Rikenʼs)Todayʼs talk
• Building a virtual large FPGA
• Concept 1: Use middle-range FPGAs and common serial links• Concept 2: Virtualize at the level of HLS description• Concept 3: Couple accelerators and a switch tightly in an FPGA
→ Accelerator-in-Switch• Our prototype: FiC (Flow-in-Cloud)• Next step: Building a virtual heterogeneous computing system

High-end, Low-cost and Mid-range FPGAs
28nm
XilinxHigh-end

20nm

16nm

10nm

Virtex-7

Virtex-Ultrascale Virtex-Ultrascale＋

2000000LC

5541000LCIntel
High-end

Stratix-V/E/GX/GS/GT
359200ALM

XilinxMid-range

Kintex-7
480000LC

Kintex-Ultrascale
1451000LCIntel
Mid-range

Arria-IV
174000LE

Arria-10
ARM＋FPU

Xilinx
Low-cost

Artix-7360000LCIntel
Low-cost

Cyclone V/E/GX/GS/GT
301000LEXCVUXXXCVUXXP

3780000LC
Stratix-10
ARM＋FPU
Kintex-Ultrascale＋
1143000LCXCKUXXXCKUXXP

Here, we will omit “XC”.
KU085 means XCKU085.

1.Multiple middle scale FPGAs vs. a single powerful FPGA
KintexUltrascale
KU085

KintexUltrascale
KU115

VirtexUltrascale
VU440

VirtexUltrascale+
VU9P

Logic Cell (K)1088 (3.4)

1451(4.3)5541(10.6)

2586(6.69)

DSP

4100(0.9)5520(1.14)2880(20.4)6840(2.53)

BRAM(Mb)

56.9(65.4)

75.9(83.0)

88.6(664.6)

345.9(50.05)

Price ($)

3720

6297

58890

17314
Price is from digikey( ) is price for each unit.

• Most of price/resource of KU085 is the lowest.
• Two KU085s can provide 1.5 LCs of KU115 with almost the sameprice.
• Five KU085s can provide almost the same LCs of VU440 with
about 1/3 price.

Price/KLCs($)
3.5

Price/KLC($) vs. Logic Cells

243

Price/KLC is increased with
its size, because high-end
FPGAs have special facilities

20
2.5

Virtex Ultrascale+

21612
1.5

Virtex Ultrascale

81

Kintex Ultrascale+

Kintex Ultrascale

0.5400200

400

600

800

1000

1200

1400

1600
KCells1800

DSP

DSP vs. Price (X $100 ）Kintex Ultrascale hassurprisingly large
number of DSPs

8000
70006000

Kintex Ultrascale
Virtex Ultrascale+

5000
4000
3000
Kintex Ultrascale+

2000

Virtex Ultrascale

1000
00100

200300400

500

600700

X $100BRAM(Mbit) vs. Price ($100)

Mbit
300

BRAM vs. Price isalmost linear,but
Kintex Ultrascale iscost effective

250

Virtex Ultrascale+

200
150
100

Virtex Ultrascale+ has
a large memory byUltraRAM but
expensive

Kintex Ultrascale
Virtex Ultrascale

50

Kintex Ultrascale+

00100

200300

400

500

600

700

X $ 100

Cost including serial links• KU085 has the best price per resource!
• Logic cells: $3.4 for 1K LC.
• DSP: $0.9 for a DSP.
• BRAM: $65.4 for 1Mb.

• Letʼs use a number of common serial links GTH (12.5Gbps).

• Of course, faster serial links (32Gbps GTY, 58GbpsGTM) are available, butcost becomes high.

• Firefly cable ($59 for 4 links) is available.
• No drivers/receivers are needed.
• Aurora IPs from Xilinx can be used.

• Conclusion: “Using multiple middle scale FPGAs” is a cost
efficient solution.
• Open issue• Cost of switches• Operational Speed

2. Virtualization at the IP based HLS designInterface
is difficultA single FPGAFPGAFPGAAXIstream

HLS 2

HLS 2

HLS 4

HLS 1

FPGAHLS 4

HLS 1

HLS 3

HLS is originally described with a set of IPs:
Division is easy except the problem of handshake.

HLS 3

The handshake problem
FPGA2

FPGA1

Data
HLS1

ReadyHLS2• Valid signal can be omitted by checking the data arrival.
• Without a ready signal, the possibility of input FIFO overflow remains.

Overriding the handshake problem• Virtual ready wire: providing a virtual wire between the receiverand the sender.
• Direct approach but the overhead of synchronization may increase.

• Providing a required memory inside HLS module.
• Convenient for streaming processing but the HLS programmer musttake care of it.

• A pre-processor can insert delay or synchronization code
according to the evaluation results from Vivado HLS.
→ All methods require fixed latency/throughput communication.
Our approach: circuit switching with Static Time Division Multiplexing.

Vivado HLS evaluates
the number of clock cycle
in a loop.
→Such information helps
override the handshakeproblem.

An example: Streaming processing
FIFOOVFHLS 1Delay200 data /5000 clock
cycles Delay

HLS 2

HLS 1

200 data /10000 clock
cycles100 data /10000 clock
cycles200 data /10000 clock
cycles200 data /5000 clock
cycles

HLS 4

HLS 3

200 data /10000 clock
cycles

HLS 5

HLS 4

HLS 3

HLS 2

200 data /10000 clock
cycles

HLS 5

100 data /10000 clock
cycles3. Integrating switch and accelerator tightly in an
FPGA: Accelerators in Switch• What is the benefit of FPGAs compared to GPUs？
• Switching capability is much superior to that of GPUs.
• Of course, recent GPUs provide NVLinks or other powerful interface.
• However, they are only for expensive GPUs, and the function is limited.

• Various type of switches can be implemented on FPGAs.
• FPGAs are widely used for high speed switches and network interface.

• Tightly coupled switch and accelerator in an FPGA.
• Separation with Partial Reconfiguration→ Accelerator in Switch [FPL2017]

An example ofAiS (PEACH3)• Implemented as a module on
the Avalon MM bus.
• Shared memory is used for exchangeof data

Intel Stratix V• Reduction / Locally Essential Treegeneration were implemented in theAiS part.

The execution time (sec)The execution time for LET generation

The number of bodies (N)21

Todayʼs talk
• Building a virtual large FPGA
• Concept 1: Use middle-range FPGAs and common serial links• Concept 2: Virtualize at the level of HLS description• Concept 3: Couple accelerators and a switch tightly in an FPGA
→ Accelerator-in-Switch• Our prototype: FiC (Flow-in-Cloud)• Next step: Building a virtual heterogeneous computing system

Flow-in-Cloud (FiC) overview
Host CPUFiC-SW

I/O board
KCU1500

FPGAFPGAFPGAFPGASTDM switch

STDM switch

STDM switchSTDM switch

Circuit switching network

High Speed
Serial Links

STDM switch

STDM switch

STDM switch

STDM switch

FPGAFPGAFPGAFPGA

HLS modules

Flow-in-Cloud (FiC) SW Board
DDR-4 SDRAM 16Gb

FiC Network
8x4 9.9Gbps

FPGA KU085/095 FPGA

Xilinx Kintex
Ultrascale XCKU095

Application
Logic Area

Here, we call eachlink “channel”,
and a bundle of
4 channels “lane”.
A board has 8 lanes
each of which has
4 channels

STDMSwitch

HLS modules
SWControl board
Raspberry Pi 3 model B

Ethernet
DDR-4 SDRAM 16Gb

Rusberry Pi 3Control NetworkBlock Diagram of FiC

DRAM
Ethernet

PR domain
HLS module

Raspi3100MHz100MHz

Static domain
9.9GbpsXilinx
AuroraSTDMswitch
170bit

8.5Gbps x32 (4 chan. x 8 lane )Xilinx
Aurora170bit

8.5Gbps x32 (4 chan. x 8 lane )9.9GbpsSTDM (Static Time Division Multiplexing)
Input data arrive at each port
cyclically registered.

Port1

S1
S2
S3
S4

S1
S2
Port2
S3
An input register is S4
selected

Port3 to the pre-loaded
accordingtable, and transferred to the output
Port 4register.

Output data are cyclicallysent to the output port

S1S2
S3
S4
S1
S2
S3
S4

Port1
An example of4x4 with four slots
Port2Port3Port 426

STDM (Static Time Division Multiplexing)
port1 ….

Port1

Port2

Port3
Port 4

S1
S2
S3
S4
S1
S2
S3
S4

P1S3

P3S4

S1
S2
S3
S4
S1
S2
S3
S4

P2S1

P2S2

P1S3

P3S4

P2S1

P2S2….

Port1• A circuit is established between
source and destination.
• Latency and bandwidth are kept.
Port2• Latency = 55+2 x (# of slots)
clock cycles

Port3Port 427

Multicast using the STDM
Port1S1S2S3S4Port2S1
S2S3S4Port3Port 4

S1S2S3S4S1S2S3S4Multicast is doneefficiently.
Port1Multiple outputs
can receive thesame data in aspecific slot.

Port2Port3
Port 4For internal usage

The resource usage
4 switches are provided for each channel.

GT: High speed link

Enough design is remained for HLS design.

How boards should be connected?
• Any type of interconnection is OK.
• However, there are two limitations:• 4 channels are bundled into a lane.
• For HLS modules, the size should be less than four 9x9 switches.

• 4 channels in a lane are used independently.

• An HLS module has four independent ports, or four HLS modules with a port areimplemented at maximum.

Network with 8-degree→ Natural solution: 4 dimensional torus
The diameter is large.
→ Alternative: Full mesh Connected Cycles（FCC)
Dragonfly-like network but more economical.

4-DTorus: the case of 3x3x3x3=81boards
Suitable if local traffic is dominant.
Diameter is relatively large: 8２００１００００００１００２０２０１ ２０２１０１ １０２００１ ００２１０１１１１０ １１１ １１２０１１０１２
１２０１２０１２１２１２0***1***

２２１ ２２２
１２２０２１ ０２２2***

4x24 Full mesh Connected Cycles (FCC)
6x4

6x4…

…

2..72..710x4024 Cycles areconnectedin full-mesh

1x4x4012..7……

x4

10
2..7…

…

6x4

6x4

96 boards are connectedEstimation of the network performance
3x3x3x3 Torus

4x24 FCC

Number of boards

8194

Slots (Bitcomplement/tornado/reversal)

11Slots (All to all)

128Diameter

85Max Latency (All to all nsec)53603550

Topology Optimization for Traffic Pattern*node swap

0010213454021

3

Recursive Partition

Generated topo. with # of slots
This slide
is supported# of slots = 2
from Dr.Hu# of switches = 12= 18
and Prof.Koibuchi #Avg.of links
SW hops = 1.313# of slots = 4# of switches = 44
# of links = 78
Avg. SW hops = 2.563# of slots = 9# of switches = 118# of links = 184
Avg. SW hops = 4.543

6indirect path

052

1

3

Circuit Switching

Result: comparison with mesh# of switches reduced
by up to 56.3%

Avg. hop count Reduced
by up to 83.7%# of slots = 12
# of switches = 606# of links = 1063
Avg. SW hops = 5.329[*] Yao Hu, Tomohiro Kudoh, Michihiro Koibuchi, "A Case of Electrical Circuit Switched Interconnection Network for Parallel
Computers", The 18th International Conference on Parallel and Distributed Computing, Applications and Technologies (PDCAT’17), pp.
276-283, Taipei Taiwan, December 18-20, 2017.

Partial Reconfiguration for separating HLS from switches.
Switch/Link synthesis

HLS module 1 synthesis
ReadSetting the PR region

opt_design, place_designroute_design

HLS1 bitmap generation

Design of HLS modules can be done
only with this part.

HLS module 2
synthesis

HLS module 3
synthesis

Make PR region into blackbox
Lock wires in the static region
Basic Design

Read into the blackbox
opt_design,
place_designroute_design

opt_design,
place_designroute_design

HLS2 bitmap
generation

HLS３ bitmap
generation

PR region for HLS module
Now, 3 lanes (12 channels)
are used.

The current FiC system
Internet /IntranetSWSWFPGAResultsControlServer
FPGA Configuration
and Table information(*.json)

SWFPGASW

GUI control from remote terminals

FPGAUsers

Deliver of Configuration
data with RESTful API
through HTTP
FiC Board Control
NetworkSW

FPGAFiC Network37

GUI from remote terminals
File configured

Now underconfiguration

Bias buffer

register・・・

Input buffer (x2)bias[0]input[0]PE[0]Input[1]・・・

Implementation
Example(LeNet Fully
Connected layer)M

input[M-1]bias[N-1]

registerPE[1]weight[0][0]weight[0][1]

・・・・
・・

weight[0][1]N×M

output[0]output[1]N

PE[N-1]weight[N-1][M-1]

Size (N, M) multiply-add・・・・・・・・・

Weight buffer

Output buffer(x2)output[N-1]

ReLu

Fully connection layer of the Lenet
Frequency（MHｚ）

Power（W)image/sec

GOPS

GOPS/W

1 board

100

17.89

23551

120.58

6.74

4 boards

100

71.56

94226

482.34

6.74

Frequency（MHz）

Power（W)

GOPS

GOPS/W

FiC 4boards

100

71.56

482.34

6.74

Stratex-V [1]

120

25.8

136.5

5.29

KCU060

200

25.0

172.0

6.88

[2]

Higher performance is achieved with the similar power efficiency compared witha single FPGA system.
[1] N.Suda, V.Chandra, G.Dasika, et.al “Throughput-optimized open-based FPGA Accelerator for large-scale
comvolutional neural networks,” FPGA2016.
[2] C.Zhang, Z.Fang, P.Zhao, P.Pan, J.Cong, “Caffeine: Towards uniformed representation and acceleration
for deep convolutional neural networks,” ICCCAD2017.

Todayʼs talk
• Building a virtual large FPGA
• Concept 1: Use middle-range FPGAs and common serial links• Concept 2: Virtualize at the level of HLS description• Concept 3: Couple accelerators and a switch tightly in an FPGA
→ Accelerator-in-Switch• Our prototype: FiC (Flow-in-Cloud)• Next step: Building a virtual heterogeneous computing system

Next Step:
Accelerator bare-metal cloud
FPGASensors

FiC NetworkData

Server Network

Data
FPGAFPGAGPUGPUGPU
GPU
MEM
MEM
MEM
MEM

FiC system

FPGA

FLOW-OS is now under development by the national institute ofindustrial science and technology (AIST)

DataSensors

Server Network

Data

GPUGPUMEM
MEMFPGAGPUFPGAGPUGPUGPUFPGAConclusion●●A virtual large FPGA:○

Scalable performance with○similar power/performance and○smaller cost/performance compared to conventional FPGA-in-clouds.

Future direction→ Accelerator bare metal cloudIntegration of FPGAs and GPUs.