An open architecture for Hardware Acceleration in ROS 2

Framework
Easily leverage hardware acceleration in a ROS-centric manner and build custom
compute architectures for robots, or robot cores.

1Hardware
Accelerationin Robotics
Perception2

Acceleration Robotics:
Understanding robotics
computations in CPUCPUTOOLSMATERIALS

Refined workshops that can build pretty muchanything, each workshop is a core

INSPECTION

Work sequentially, task after task
until finalizing
Multiple workers or,“threads”, on each CPU (though it
eventually fills the space)

ASSEMBLY

Modern CPUs have various coresWORKSHOPSOLDERING

3

Acceleration Robotics:
Understanding robotics
computations in GPUs
MATERIALSGPU

Many workers doing repetitive work
START SHIFTExpert, designs masterplan (hard to find, few)
TOOLS

Not power eﬃcient
END SHIFT

GPUs have many industrial workshops

INDUSTRIAL WORKSHOP

4

Acceleration Robotics:
Understanding robotics
computations in FPGAs
EXPERT ARCHITECT 1

FPGA
Flexible and adaptable factory (software defined)
EXPERT ARCHITECT 2Expert-driven architectures
Low power, high performance

RECONFIGURATION

ADAPTABLE FACTORY

5

Acceleration Robotics:
Understanding robotics
computations in ASICs

ASIC

Fixed, fully automated factorySpecialized in one process
Very expensive to design and build
Best throughput at lowest power consumption

FIXED FACTORY6ROS 2 Hardware Acceleration Working Group (HAWG)The ROS 2 Hardware Acceleration Working Group drives the creation,
maintenance and testing of acceleration kernels on top of open standards for
optimized ROS 2 and Gazebo interactions over diﬀerent compute substrates(including FPGAs, GPUs and other accelerators).

ß accelerateddataflow

π dataflow

https://github.com/ros-acceleration/communityt(π)

1xt(ß) << t(π)

1x - 100x

6350

ACCELERATORVENDORSREGISTEREDATTENDEESPARTICIPATING

HAWG#117

ROBOTCORE Framework:
An open architecture for Hardware
Acceleration in ROS 2Robotics Application

CPUin the form of acomputational graphCommon ROS 2 API

Hardware Acceleration Framework for ROS
It helps build custom compute architectures
for robots through acceleration kernels, orrobot cores, that make robots faster, moredeterministic and power-eﬃcient. Simply
put, it provides a development, build anddeployment experience for creating robot
hardware and hardware acceleration
kernels similar to the standard, nonaccelerated ROS development flow.

Open Architecture for Hardware Acceleration in ROS 2
CPUFPGACPUFPGACPUGPU

Integrated Tracing and Benchmarking

Default ROS 2Option 1

CPUOption 2

Option 3…

…
FPGACPUGPU

Option N

https://arxiv.org/pdf/2205.03929.pdf

8ROBOTCORE Framework
Hardware Acceleration Framework for ROS. It helps build custom computearchitectures for robots, or robot cores, that make robots faster, more deterministic andpower-eﬃcient. Simply put, it provides a development, build and deployment experiencefor creating robot hardware and hardware accelerators similar to the standard, nonaccelerated ROS development flow.

Public community-driven and open source
implementation available at https://github.com/ros-acceleration

ß accelerateddataflow
π dataflowt(ß) << t(π)t(π)

Framework
1x

1.5x - 100x9

ROBOTCORE Framework:
Production-grade multi-platform ROS
support with Yocto (REP 2000 ➦)Instead of relying on common
development-oriented Linux distros(such as Ubuntu), our contributions toYocto allow to build a customized Linuxsystem for your use case with ROS,
providing unmatched granularity,performance and security.

KR260

KV260

ZCU102ZCU104

Jetson Nano

Jetson Xavier NX

Jetson AGX Xavier

PolarFire Icicle

10From ROBOTCORE Framework to an open standard
ROS 2 Hardware Acceleration Architecture andConventions (REP 2008 ➦)We are bringing the lessons learned
while developing ROBOTCORE
Framework into a REP that
describes the architectural pillars
and conventions required tointroduce hardware acceleration inROS 2 in a vendor-neutral, scalable
and technology-agnostic manner.

APPLICATION

ROS 2

All while maintaining the common
ROS development flow.

DDS

UDP/IPROBOTCORE Framework

AMD

NVIDIAMICROCHIP(Other silicon architecture)11

depicts the most popular inspired by [5]:From ROBOTCORE Framework to an open standard
Benchmarking performance in ROS 2 (REP 2014 ➦)Figure 1

We adopt a grey-box and nonfunctional benchmarking approachfor hardware acceleration with a lowoverhead tracing and benchmarking
framework, and select the Linux
Tracing Toolkit next generation (LTTng)

Performancebenchmarking
approaches.
Functional(top-left),
Non-functional(top-right),
Black-Box(bottom-left) andGrey-box(bottom-right).

Probe+++++++v- latency- throuhput- memory- CPUv
Function++System under test+

System under test+++FUNCTIONAL+343ʺ+:3(9.43&1+++Application+v vv+v++ Probe

System under test+'1&(0ʺ'4=+ Probev

System under testv

Probes++ Probev+++Test App.
+ + + +

to implement it.

+ Probe+ Probe+ Probe+ Probev v v v

Benchmarking is the act of running acomputer program with a known
workload to assess the program'srelative performance.

Probe+++,7*>ʺ'4=12

Case Study: ROS 2 Perception graph
ROBOTCORE Perception is an optimized robotic perception
stack built with ROBOTCORE Framework that leverages
hardware acceleration to provide a speedup in your perception
computations. API-compatible with the ROS 2 perception stack,
ROBOTCORE Perception delivers high performance, real-timeand reliability to your robots' perception.

PERCEPTIONSUBGRAPHßaccelerated
dataflowπdataflow+→
ROBOTCORE
Framework

t(π)

1x

ACCELERATEDPERCEPTIONSUBGRAPHPERCEPTIONSUBGRAPHµaccelerated
dataflow+t(ß) < t(π)→
ROBOTCORE
Perceptiont(µ) << t(π)

5-100x13PERCEPTIONSUBGRAPHπdataflow

rectify

ROS 2 Perception graph: 2 nodes perception pipeline

1xresizet(π)+ROS 2 perception pipelinekernel runtime latency (ms)240180ROBOTCORE+

3.5x12060

ROBOTCORE
Perception→0CPU + GPU

CPU + FPGA

µaccelerated
dataflow

ACCELERATEDPERCEPTIONSUBGRAPH

rectify

ROS 2 perception pipeline (2 nodes) - kernel runtime latency (ms). Measured ROBOTCORE Perception pipeline runtime latency running on an AMD KV260 versus
NVIDIA Isaac ROS running in a Jetson Nano 2GB. Measurements present the kernel runtime in milliseconds (ms). Source code available at perception_2nodes.

3.5x

resizet(µ) << t(π)

14PERCEPTIONSUBGRAPH

πdataflow

rectify

ROS 2 Perception graph: 2 nodes perception pipeline

1xresize
t(π)

+2,25

1,50,75

ROBOTCORE+6x

ROBOTCORE
Perception→

ROS 2 perception pipeline
Performance-per-watt (Hz/W)

30

CPU + GPUCPU + FPGA

µaccelerated
dataflow

ACCELERATEDPERCEPTIONSUBGRAPH

rectify

ROS 2 perception pipeline (2 nodes) - Performance-per-watt (Hz/W). Measured ROBOTCORE running on an AMD KV260 versus NVIDIA Isaac ROS running in a Jetson Nano 2GB.
Measurements present a 2-Node (rectify and resize operations) pre-processing perception graph performance-per-watt (Hz/W). Source code available at perception_2nodes.

6xresizet(µ) << t(π)

15Case Study: ROS 2 Perception Nodes

πdataflowßaccelerated
dataflow

HarrisHarris++→t(π)

ROBOTCORE
Framework

1xROBOTCORE
Perceptiont(ß) << t(π)

30x16

π
dataflow

ROS 2 Perception Nodes: Harris perception NodeHarris

1xt(π)+Harris perception Nodekernel runtime latency (ms)160120ROBOTCORE+80

30.27x40ROBOTCOREPerception→0CPU + GPU

CPU + FPGAßaccelerated
dataflow
Harris30xHarris - kernel runtime latency (ms). Measured ROBOTCORE Perception running on an AMD KV260, NVIDIA Isaac ROS running in a Jetson Nano 2GB. Measurements
present the kernel runtime in milliseconds (ms) and discard ROS 2 message-passing infrastructure overhead and host-device (GPU or FPGA) data transfer overheadt(ß) << t(π)

17πdataflow

ROS 2 Perception Nodes: HOG perception NodeHOG

1xt(π)+HOG perception Nodekernel runtime latency (ms)500375ROBOTCORE+250509.5x

125

CPU + GPUCPU + FPGA→0ROBOTCORE
Perceptionßaccelerated
dataflow
HOG

Histogram of Oriented Gradients (HOG) - kernel runtime latency (ms). Measured ROBOTCORE Perception running on an AMD KV260, NVIDIA Isaac ROS running in a Jetson Nano 2GB.
Measurements present the kernel runtime in milliseconds (ms) and discard ROS 2 message-passing infrastructure overhead and host-device (GPU or FPGA) data transfer overhead

509xt(ß) << t(π)

182022
Hardware Acceleration
Report in Robotics
Captures the state-of-the art of
hardware acceleration in robotics byfollowing a quantitative approach andpresents robotic architects with aresource to consider while designing
their robot computational architectures.
First phase, a community survey
conducted in both the ROS andthe overall robotics communities
helped grasp the interest behindthe use of hardware accelerationin robotics. Input from thiscommunity survey was then used
to drive theSecond phase, a hardware
acceleration benchmarking
eﬀort

19

2022 Hardware Acceleration
Report in Robotics
First phase: We’re pushing forward REP-2008
initiative to better integrate hardware
acceleration with ROS and Gazebo,what’s mostimportant for you? (report➦)

96 answers

20

DEFG

1 (1%)C

1 (1%)B

1 (1%)A

1 (1%)

0
when
itcomestohardware
accelerating
ROS
2and/orGazebo?
(report➦)96 answers (multiple answers allowed)

1 (1%)



What packages/components
2022
Acceleration
doHardware
you think
we should prioritize
Reportin Robotics
when
it comes to hardware
accelerating
ROS
2and/orGazebo?
First phase: What packages/components do you think weshould prioritize

20HI

JKL

80

1 (1%)

1 (1%)

1 (1%)

1 (1%)

1 (1%)

20

19 (19,8%)20 (20,8%)62 (64,6%)
21 (21,9%)40

29 (30,2%)60

39 (40,6%)

from theare accein ROS 2o survey”
question:ackages/nents do
we should
e when ithardware
ng ROS 2Gazebo?”.

58 (60,4%)

Figure 6

A. Gazebo/Ignition physic engines
B. 748SF[NLFYNTSXYFHP SF[NLFYNTS
C. 748RFSNUZQFYNTSXYFHP 2T[J.Y
D. ROS 2 perception stackE. 748HTRRZSNHFYNTSRNIIQJ\FWJ ))8NJTʈTFINSLNYYTMFWI\FWJ
F. 748SJY\TWPNSLXYFHP :)5.5*YMJWSJYRTWJIJYJWRNSNXYNHSJY\TWPNSYJWFHYNTSXG. ROS 2 control stackH. &QQTKYMJRFWJNRUTWYFSY&HHJQJWFYNSL,F_JGTHTZQIGJZXJKZQ\MJS\TWPNSL\NYM
X^SYMJYNHJS[NWTSRJSYKTW71TW)719MJTYMJWFWJGTYMKTWYNRNSLXFSIIJYJWRNSNXYNH
properties of the nodes
I. Webots physics engine
J. .RFLJFSIIJUYMIFYFUWTHJXXNSLUNUJQNSJX.RUWT[JRJSYXTSNRFLJFSIIJUYMIFYF
HTRUWJXXNTSFSIYMJNWNSYJLWFYNTS\NYMWTXGFLWJHTWINSL
K. Lidar drivers and perception

HIJKL0A

B

C

DE

F

Ganswers
A. Gazebo/Ignition96
physicengines(multiple answers allowed)
26
B. 748SF[NLFYNTSXYFHP SF[NLFYNTS
C. 748RFSNUZQFYNTSXYFHP 2T[J.Y
D. ROS 2 perception stackE. 748HTRRZSNHFYNTSRNIIQJ\FWJ ))8NJTʈTFINSLNYYTMFWI\FWJ
F. 748SJY\TWPNSLXYFHP :)5.5*YMJWSJYRTWJIJYJWRNSNXYNHSJY\TWPNSYJWFHYNTSXL. I’d like to see more general tools that can be implemented as nodes or library calls thatFQQT\RJYTVZNHPQ^GZNQIFHHJQJWFYJIFQYJWSFYN[JXKTWR^X^XYJR

21

hardware acceleration?
comes to hardware
acceleration?”.

2022 Hardware Acceleration
94 answers
Report in Robotics

48,9%First phase: What do you care more
about when it comes to hardware
acceleration? (report➦)

Speed (or latency):
time between
the start and thecompletion of atask

46,8%Real-time: Meetingtime deadlines intheir computations
Determinism: thata task happens inexactly the same
timeframe eachtime

48,9%

94 answers

Bandwidth orthroughput(the total amount of work done in agiven time for a task)

Speed

Speed (shorterJ]JHZYNTSYNRJ

49 %(shorter execution time
between the start and thecompletion of a task)

Real-time

Real-time anddeterminism

11 %(ability of completing a task's
computations while meeting timedeadlines)

28 %Power consumption

12 %

Power
(the electrical energy per unit of time
consumed while executing a giventask)

98 answers

22

Speed (shorterPERCEPTIONSUBGRAPH

πdataflow

2022 Hardware Acceleration
Report in Robotics

rectifyresize

Second phase: Benchmarking hardware acceleration (report➦)t(π)+
ROS 2 perception graph performance-per-watt with hardware acceleration (Hz/W)

ure 0-permark
OS 2graphrious
s. Thegraph
ribed+

2
1.5
1

ROBOTCORE
Perception→

n 4.2.
etter.

ROBOTCORE

2.5

0.50
CPU + FPGA (KR260)

CPU + FPGA(Jetson Nano)

CPU + GPU(Jetson AGX Xavier)µaccelerated
dataflowACCELERATEDPERCEPTIONSUBGRAPH

rectify

resizet(µ) << t(π)

ROBOTCORE tools and robot cores
ROS 2 API compatible hardware acceleration tools androbot Intellectual Property (IP) cores. Increasing yourrobot's performance, including latency, power eﬃciency
and platform scalability.

ROBOTCORE®
Framework

ROBOTCORE®
Cloud

ROBOTCORE®
Perception

ROBOTCORE®
Transform

Hardware acceleration
framework for
ROS and ROS 2.

Speed-up ROS 2
graphs with/inthe cloud.

Accelerated ROS 2
perception
stack.

Accelerated
ROS 2 coordinatetransformations (tf2).

ROBOTCORE®
******Accelerated
ROS 2 ****:24FogROS2 and ROBOTCORE Cloud: Tools to speed-up ROS 2 graphs with the cloud, and in the cloud.

COMPUTATIONALLY
EXPENSIVESUBGRAPHßaccelerated
dataflow

πdataflow+→
ROBOTCORE
Framework

t(π)

1x

COMPUTATIONALLY
EXPENSIVE SUBGRAPHCOMPUTATIONALLY
EXPENSIVESUBGRAPH+t(ß) << t(π)

1.5x

→

µaccelerated
dataflow

COMPUTATIONALLY
EXPENSIVE SUBGRAPH+

ROBOTCORE
Cloudt(µ) << t(π)

10xhttps://github.com/BerkeleyAutomation/FogROS2→∂accelerated
dataflow

ROBOTCORE
Perceptiont(∂) << t(π)

1000xRobotics MCU: A robotics microcontroller unit (MCU) powered by RISC-V and ROS 2100 Mbps

1x EthernetmW

ROS 2budget applications

Target: Iron Irwini (May ’23)

OMG DDS
and RTPS
interoperability
DDS v1.4 and RTPS 2.3

More peripherals
2xUART, 2xI2C master, 1xI2C slave,
2xQSPI master, 1xSDIO

Robotics
MCU

32-bitRISC-V ISA

1-corein-order, 4pipeline-stage

4x PWM
Silicon-proven
Nodes from 130 nm to 22nm

Apache 2.0
and MIT
Commercially-friendly licenseenergyeﬃcient

Best-eﬀort and reliable
QoS policies available

512 KBSRAM memory

27

ROBOTCORE:
The Robotic Processing Unit specialized in ROScomputations ➦ROBOTCORE® is a robot-specific processing unit that helps map
Robot Operating System (ROS) computational graphs to its CPUs,
GPU and FPGA eﬃciently to obtain best performance. It empowers
robots with the ability to react faster, consume less power, anddeliver additional real-time capabilities.

https://github.com/ros-acceleration/robotic_processing_unit

28

Thanks
Q&A29