2017/9/22

ROSCon2017@Vancouver

1ROS-COMPLIANT FPGA COMPONENT TECHNOLOGY– INSTALLATION OF FPGA INTO ROS
Takeshi Ohkawa*, Yutaro Ishida**,Yuhei Sugata*, Hakaru Tamukoh***Utsunomiya University,**Kyushu Institute of Technology
This research and development work (done by Utsunomiya Univ.)
was supported by the MIC/SCOPE#152103014.

PEAR-LAB Utsunomiya Univ. / Tamukoh Lab. Kyutech

2017/9/22

2

ROSCon2017@Vancouver

Problem of computer vision on robots

RoboCup 2016
Domestic Pre-Competition× 5 Speed

Over 1 min. to recognize just six objects
PEAR-LAB Utsunomiya Univ. / Tamukoh Lab. Kyutech

2017/9/22

3ROSCon2017@Vancouver

FPGA accelerates DNN• “1 min. is too long for the task!”• GoogLeNet[3] (22 layers)• 4 seconds for an inference• CPU: Core i5-5200U 2.2GHz• Need of accelerator• GPU – Power consumption, Heat!
• FPGA – optimized circuit for the app = high performance/power efficiency[3] “Model Zoo”, http://dl.caffe.berkeleyvision.org/bvlc_googlenet.caffemodel
PEAR-LAB Utsunomiya Univ. / Tamukoh Lab. Kyutech

2017/9/22

4ROSCon2017@VancouverEvaluation of CPU, GPU, FPGA*・Model of Deep Neural Networks: VGG16・Dataset: Cifar 10
Xilinx ZCU102(FPGA)NVIDIA Jetson TX1Device
Clock Freq.
Efficiency [FPS/W]

Accuracy [%]Quad-core
ARM Cortex-A57

256-core
Maxwell GPUZynq UltraScale+
MPSoC

1.9 GHz

998MHz

100MHz

0.032

0.376

1.43

92.35

90.30
3.8 times efficient* H. Yonekawa and H. Nakahara, “On-Chip Memory Based Binarized Convolutional Deep Neural Network Applying Batch Normalization Free Technique on an
FPGA,” Parallel and Distributed Processing Symposium Workshops (IPDPSW), 2017 IEEE International, 2017.
PEAR-LAB Utsunomiya Univ. / Tamukoh Lab. Kyutech

2017/9/22

ROSCon2017@Vancouver

5Recent trends on FPGA• Application• Image Recognition (local feature, DNN)
• Security (encryption)• Big Data (compression, mining)• Device• Single Chip - ARM + FPGA•••

Mar. 2011, Xilinx Zynq
Oct. 2011, ALTERA SoC FPGAIntel's $16.7B Altera Buyout(June 2, 2015)2016- (announced) Intel Broadwell Xeon + FPGA (ALTERA Arria 10)• Tool• High-Level Synthesis (C to HDL)
• Service• Apr. 2017, Amazon EC2 F1 instance•FPGA: Xilinx Virtex UltraScale+ VU9P（2,586,150 Logic Cells)
PEAR-LAB Utsunomiya Univ. / Tamukoh Lab. Kyutech

2017/9/22

6

ROSCon2017@Vancouver

Problem for introducing FPGA
• “Difficult design of FPGA”• Why FPGA is difficult to develop?
• Language: HDL (Hardware Description Language) or HLS (High-level Synthesis)• Need of performance tuning techniques (processing and memory access: FPGA expertise)• Long compile time

• Component technology is necessary!
• In order to use/reuse high-performance FPGA design easily from software.
• Let’s make ROS-node as FPGA-component!

PEAR-LAB Utsunomiya Univ. / Tamukoh Lab. Kyutech

2017/9/22

7ROSCon2017@Vancouver

Our proposal: FPGA component technologies for ROS system (3 types)• (1) COMTA (PC andProgrammable SoC*)• (3) Fully-hardwired• (2) ROS-compliant• PC: ROS on LinuxFPGA component on
Programmable SoC*

• ARM: bridge (TCP/IP)• ARM: ROS on Linux• FPGA: Application logic• FPGA: Application logic

ROS-compliant FPGAcomponent• Talks ROS protocol byusing TCP/IP stack• Direct connection toFPGA Application logic

ROS-Compliant FPGA Component(SoC)
Publisher

input

PS(ARM)
Subscriber
Publisherinterfacefor FPGA

interfacefor FPGAROS node

Subscriber

outputPublisher

input

Hardware ROS-Compliant FPGA Component
FPGA

Subscriber
HW

Application

Publisher
HW

Subscriber

outputPL(FPGA)

Application

Proposed by Kyutech
Mar. 2015

Proposed by Utsunomiya Univ.
Mar. 2015

ROS node

Proposed by Utsunomiya Univ.
Mar. 2017*Programmable SoC: System-on-chip, ARM+FPGA, i.e. Xilinx Zynq-7000, Intel (ALTERA) SoC FPGAPEAR-LAB Utsunomiya Univ. / Tamukoh Lab. Kyutech

2017/9/22

8ROSCon2017@Vancouver(1)COMTA Experiment: Following human by using image processing

Processing results are given by custom ROS topic
PEAR-LAB Utsunomiya Univ. / Tamukoh Lab. Kyutech

2017/9/22

9ROSCon2017@Vancouver(1)COMTA Experiment: Results

Computing load of Conv. method

Status: Idling
DeviceFrame Rate
[FPS]
Power
[W]
Efficiency[FPS/W]

Conv.

Prop.

Core i55200U
2.2GHzARM Dual Core
667MHz + FPGA

30%

29.06

17.25

26

4.7

1.12

3.69

3.3 times efficient

50 points increase
Status: Following human80%

Heavy task can beoffloaded onto FPGA!
PEAR-LAB Utsunomiya Univ. / Tamukoh Lab. Kyutech

2017/9/22

ROSCon2017@Vancouver(2) ROS-compliant FPGA component on Programmable SoC
cReComp: automatic generation of ROS component form HDL• Target*.scrp or *.py

cReCompconfiggenerate

SW I/FROS node*.cpp
Software

HW I/F
FIFO
FIFO

Data comm.

Userlogic*.v*.v
Hardware

ROS-Compliant FPGA component

10• Programmable SoC (Xilinx Zynq)• Input• Application Logic: HDL• Configuration Fie:
• scrp (Text)

Python (using PyVerilog)

Zedboard• Output• HDL: FIFO buffer control• C++: ROS nodeExperimental (6 undergraduate/graduates）result:finished componentization within an hour! (42 minutes maximum)

ZYBO

Shared at https://github.com/Kumikomi/cReComp
PEAR-LAB Utsunomiya Univ. / Tamukoh Lab. Kyutech

2017/9/22

11

ROSCon2017@Vancouver(2) ROS-compliant FPGA component on Programmable SoC
Open-Source Packages Available in ROS.org

Shared athttps://github.com/Kumikomi/openreroc_motion_sensor
https://github.com/Kumikomi/openreroc_pwm
PEAR-LAB Utsunomiya Univ. / Tamukoh Lab. Kyutech

2017/9/22

12ROSCon2017@Vancouver(3) Fully-hardwired ROS-compliant FPGA component
DEMO MOVIE: FPGA-ROS node of FAST key-point detection
USB camera

InputImage
FAST key-points
by FPGAFAST key-points
by PC

ROS node
on FPGA board

PEAR-LAB Utsunomiya Univ. / Tamukoh Lab. Kyutech

2017/9/22

13ROSCon2017@Vancouver(3) Fully-hardwired ROS-compliant FPGA component- structure of the evaluation system• FPGA is accessed via Gigabit Ethernet
FPGA• Image processing circuit
Sub HW• HLS (Xilinx VivadoHLS) from C++ (OpenCV)• Subscriber/Publisher HW (HDL)• Implemented using TCP/IP stack (SiTCP*)
image_view
uvc_camera_
node

image/raw

pre_process_node/input_data

IP:192.168.10.17

Imageprocessingcircuit(AXI-Stream)

Pub HW

IP:192.168.10.18

Fast_PC*T.Uchida: “Hardware-Based TCP Processor for Gigabit Ethernet," IEEE Transactions on Nuclear/output_data

image_view/output_pc

image_view

Science,Vol.55, No.SIG 3, pp.1631-1637, 2008.

PEAR-LAB Utsunomiya Univ. / Tamukoh Lab. Kyutech

2017/9/22

14

ROSCon2017@Vancouver(3) Fully-hardwired ROS-compliant FPGA component
Source code of Image Processing Circuit – for Xilinx Vivado HLS

PEAR-LAB Utsunomiya Univ. / Tamukoh Lab. Kyutech

2017/9/22

15

ROSCon2017@Vancouver

How does the FPGA ROS-node (Hardware Pub/Sub) work?
• ROS wire protocol (TCP/IP packets) i.e. Publish/Subscribe messages• Anyone can be a ROS node who talks TCP/IP in ROS manner.
• It is not the matter if it is Software, Hardware (FPGA) or something!
NodeNodemsg

Publication

Topic

NodeSubscription

SoftwareHardware (FPGA)

somethingLet’s hack the ROS wire protocol.
PEAR-LAB Utsunomiya Univ. / Tamukoh Lab. Kyutech

2017/9/22

16

ROSCon2017@Vancouver

Procedure of Publish/Subscribe messaging in ROS system (1/3)
master
①advertise(“bar”, foo:1234)

XMLRPC①subscribe(“bar”)②{foo:1234}

Publisher

Subscriber• STEP① and STEP② is communication with master• master is a name server in ROS system• XMLRPC: remote procedure calling using HTTP as thetransport and XML as the encoding

PEAR-LAB Utsunomiya Univ. / Tamukoh Lab. Kyutech

2017/9/22

17

ROSCon2017@Vancouver

Procedure of Publish/Subscribe messaging in ROS system (2/3)
XMLRPCmaster

①advertise(“bar”, foo:1234)①subscribe(“bar”)②{foo:1234}③requestTopic(“bar”,TCP)

Publisher④

foo:2345

Subscriber• STEP③: Subscriber sends a connection request

to Publisher by using requestTopic (XMLRPC)• STEP④: Publisher returns host address andport number of TCPROSPEAR-LAB Utsunomiya Univ. / Tamukoh Lab. Kyutech

2017/9/22

18

ROSCon2017@Vancouver

Procedure of Publish/Subscribe messaging in ROS system (3/3)
XMLRPCTCPROS

master①subscribe(“bar”)①advertise(“bar”, foo:1234)②{foo:1234}③requestTopic(“bar”,TCP)
Publisher

Subscriber④ foo:2345⑤ connect(foo:2345)⑥data transmission• Binary data transmission starts in TCPROS protocol• STEP⑤: Subscriber establishes a TCP connection• STEP⑥: Data transmission repeats
PEAR-LAB Utsunomiya Univ. / Tamukoh Lab. Kyutech

2017/9/22

19

ROSCon2017@Vancouver

TCPROS protocol• Binary data: efficient data transfer• Easy to handle in hardware• Example:• ROS message = int32 x2

ROS Message

int32
int32ab

TCP/IP packet (captured by WireShark)

52 54 00 60 81 a8 52 54 00 26 a6 ae 08 00 45 00
00 40 fb 14 40 00 40 06 c9 57 c0 a8 7a 73 c0 a87a 87 8a 81 df 36 8f de ee 08 61 a8 82 70 80 1800 eb 76 7e 00 00 01 01 08 0a 00 03 19 ba 00 02
de 55 08 00 00 00 04 00 00 00 05 00 00 00
Data: message size(int32) +int32+int32
PEAR-LAB Utsunomiya Univ. / Tamukoh Lab. Kyutech

2017/9/22

20ROSCon2017@Vancouver(3) Fully-hardwired ROS-compliant FPGA component- Separation for Hardwired Publisher NodePublisher SW
topic name: “bar”hostname: foo
XMLRPC port number: 1234
Publisher HWhostname :FPGATCPROS port number: 3456
Subscriber
subscribe to “bar”
XMLRPCTCPROS

master①subscribe(“bar”)①advertise(“bar”, foo:1234)

XMLRPCTCPROS

master①subscribe(“bar”)①advertise(“bar”, foo:1234)②{foo:1234}

Publisher

Publisher③requestTopic(“bar”,TCP)④ FPGA:3456

Subscriber

SW on PC②{foo:1234}③requestTopic(“bar”,TCP)
Publisher④ foo:2345
Subscriber⑤ connect(foo:2345)

Publisher
HW on FPGA

⑥ data transmission• Publisher HW can be implemented with 1 TCP/IP port!
PEAR-LAB Utsunomiya Univ. / Tamukoh Lab. Kyutech

2017/9/22

21

ROSCon2017@Vancouver(3) Fully-hardwired ROS-compliant FPGA component- Evaluation system• Measurement: communication latency• FPGA component does not process data (relay only)• Used ROS message type: “sensor_msgs/image”• Comparison: PC/ARM/FPGAPublish

Subscribe Publish

output_data

input_data

msg
input

Subscribe

PC/ARM/FPGA

outputComp.

latencyPEAR-LAB Utsunomiya Univ. / Tamukoh Lab. Kyutech

2017/9/22

22

ROSCon2017@Vancouver(3) Fully-hardwired ROS-compliant FPGA component- Amount of hardware resource• FPGA: Xilinx Spartan-6, XC6SLX110
Board: exTri-CSI (e-trees.Japan, Inc.)*• Two SiTCP** instances for Gigabit Ethernet ports•*FPGA does not process data (relay only)

RESOURCEUTILIZATION

Slice Registers

9253/126576 ( 7%)

Slice LUTs

7421/63288 (11%)

RAMB16BWERs* http://e-trees.jp/products/extri-csi

54/268 (20%)**T.Uchida: “Hardware-Based TCP Processor for Gigabit Ethernet," IEEE Transactions on Nuclear
Science,Vol.55, No.SIG 3, pp.1631-1637, 2008.

PEAR-LAB Utsunomiya Univ. / Tamukoh Lab. Kyutech

2017/9/22

23

ROSCon2017@Vancouver(3) Fully-hardwired ROS-compliant FPGA component- Evaluation results: Image data transfer (sensor_msg/image)350
300time(ms)

250
200

290

Our implementation is fasterthan software on ARM processor
129

150

101

100
50

2555

63

48

34

175

01M

2M

6M

data size(byte)
PC-PC

PC-ARM

PC-FPGA• Throughput: 550Mbps• 58% of maximum performance (SiTCP’s MAX 949Mbps)
PEAR-LAB Utsunomiya Univ. / Tamukoh Lab. Kyutech

2017/9/2224

ROSCon2017@VancouverSummary & Future work
• Proposal: Three types of ROS-compliant FPGA components• Robot applications of the components• Object recognition using DNN architecture• SLAM: Distributed Visual-SLAM processing• Robot: Image processing (ORB feature) on FPGA
• Cloud: SLAM processing at highly-parallel environment• Challenges at RoboCup@Home2018!
• Movie: RoboCup@Home2017
1st place Winner! at domestic standard
platform league (TOYOTA HSR)
PEAR-LAB Utsunomiya Univ. / Tamukoh Lab. Kyutech

2017/9/22

ROSCon2017@Vancouver

25

THANK YOUPEAR-LAB Utsunomiya Univ. / Tamukoh Lab. Kyutech

2017/9/22

26

ROSCon2017@Vancouver

References• POINTERS to Packages:• OpenReroc (Open source Reconfigurable Robot Component) project, which aims topromote the concept of ROS-compliant FPGA component• http://kumikomi.github.io/OpenReroc/• Example implementation of Xilinx Zynq (ARM+FPGA) for Indigo packages at ros wiki
• http://www.ros.org/browse/details.php?distro=indigo&name=openreroc_pwm• http://www.ros.org/browse/details.php?distro=indigo&name=openreroc_motion_sensor• URLs:• Home service robot project team “Hibikino-Musashi” for RoboCup@Home
http://www.brain.kyutech.ac.jp/~hma/wordpress/about-us/• RoboCup@Home Movie https://www.youtube.com/channel/UCJEeZZiDXijz6PidLiOtvwQ• RoboCup@Home Website http://www.robocupathome.org/• Zynq-7000 All Programmable SoC, Xilinx:
http://www.xilinx.com/products/silicon-devices/soc/zynq-7000.html
PEAR-LAB Utsunomiya Univ. / Tamukoh Lab. Kyutech

2017/9/22

ROSCon2017@Vancouver

27

References (2/2)
••

•

••
•
•

[1] Takeshi Ohkawa, Kazushi Yamashina, Takuya Matsumoto, Kanemitsu Ootsu, Takashi Yokota、"Architecture Exploration ofRobot System using ROS-Compliant FPGA Component"、Proc. 27th IEEE International Symposium on Rapid System Prototyping(RSP), pp.72-78, Oct. 2016. DOI: http://dx.doi.org/10.1145/2990299.2990312[2] Yuhei Sugata, Takeshi Ohkawa, Kanemitsu Ootsu and Takashi Yokota, "Acceleration of publish/subscribe messaging in
ROS-Compliant FPGA Component," The 8th International Symposium on Highly Efficient Accelerators and Reconfigurable
Technologies (HEART2017), 7-9 June 2017 (PDF is not available yet)
http://www.ruhr-uni-bochum.de/heart2017/program.html[3] Kazushi Yamashina, Takeshi Ohkawa, Kanemitsu Ootsu and Takashi Yokota, “Proposal of ROS-compliant FPGA Component
for Low-Power Robotic Systems - case study on image processing application,” Proceedings of 2nd International Workshop onFPGAs for Software Programmers, FSP2015, pp. 62-67, 2015. https://arxiv.org/ftp/arxiv/papers/1508/1508.07123.pdf[4] Kazushi Yamashina, Hitomi Kimura, Takeshi Ohkawa, Kanemitsu Ootsu, Takashi Yokota, “cReComp: Automated Design Toolfor ROS-Compliant FPGA Component,” In proc. IEEE 10th International Symposium on Embedded Multicore/Many-core
Systems-on-Chip (MCSoC-16), 2016.
[5] A. Suzuki, T. Morie, H. Tamukoh, “FPGA Implementation of Autoencoders Having Shared Synapse Architecture,” Proc. of the23rd Int. Conf. on Neural Information Processing (ICONIP2016), Lecture Notes in Computer Science, Vol. 9947, pp.231-239, 2016.
[6] S. Yokota, J. Li, Y. Ogishima, H. Kubo, H. Tamukoh, and M. Sekine, “Self-Learning of Feature Regions for Image Recognition,”
Journal of Computer Sciences and Applications, Vol.3, No.1, pp.1-10, 2015[7] Yutaro Ishida, et al., “Approach to accelerate the development of practical home service robots - RoboCup @Home DSPL”,
The 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN 2017), 2017.
PEAR-LAB Utsunomiya Univ. / Tamukoh Lab. Kyutech