ROSCo n 2023
Using Reference System to evaluate
features and performance in astandardized and repeatable way

Lyle Johnson
on behalf of Evan Flynn
October 20, 2023
© 2023 Apex.AI, Inc. | All rights reserved.
Apex.AI, Apex.OS, Apex.Grace, Apex.Ida, Apex.Alan, Apex.Ray are trademarks of Apex.AI.

Agenda1.Filling a tooling gap
2.Execution management configuration
3.Saving time in automotive development© 2023 Apex.AI, Inc. | All rights reserved.
Apex.AI, Apex.OS, Apex.Grace, Apex.Ida, Apex.Alan, Apex.Ray are trademarks of Apex.AI.

2

Filling a tooling gap

A Big Bang of executor modes
led to the creation of theReference System tool.

© 2023 Apex.AI, Inc. | All rights reserved.
Apex.AI, Apex.OS, Apex.Grace, Apex.Ida, Apex.Alan, Apex.Ray are trademarks of Apex.AI.

1Apex.AI developed a custom
executor. ROS 2 was adding afew executor modes.

2

Needed a fair and standard way
to measure the executor’sperformance, both against itself
and other implementations

3Xyvr w$syx$q i ewyvmr k$t i vjsvq er gi $mw$jemvp} $l evh$Ħ xss$q er } $w} wxi q $gsr jm
kyvexmsr w0$i | i gyxsv$
gsr jmkyvexmsr w0$er h$OTM
w2

4

Need a standardized, repeatable
test fixture that can deliver threethings:© 2023 Apex.AI, Inc. | All rights reserved.
Apex.AI, Apex.OS, Apex.Grace, Apex.Ida, Apex.Alan, Apex.Ray are trademarks of Apex.AI.

Motivation

Wha t le d to c re a ting Re fe re nc e Sys te mA. Measure performance KPIs like latency,
jitter, CPU, and memory usage
B. Flexibility to allow for different executors
and various executor configurations whilestill keeping the overall system the same
C.And tooling to automate running experiments
and coordinating test campaigns

4Motivation

Creating the Reference System

52[ mxl $xl swi $vi ewsr w$m
r $q mr h0$xl i $jm
vwx$m
xi vexmsr $sj$xl i $Vi ji vi r gi $W} wxi q ${ ew$gvi exi h$jsv$VSW$[ svph$6465${ mxl $xl i $gsppi gxm
zi $i jjsvx$sj$xl i $VSW$Vi ep1Xm
q i $[ Ke2Wl syx$syx$xs$Vept l 0$Ner 0$Er hvi m0$L } swi yr k0$QT0$Gl vm
wxm
er 0$Gl vm
wxst l i 0$er h$Gevpsw%
62Xl mw${ ew$er $e{ i wsq i $hm
wt pe} $sj$xl i $gsq q yr mx} $gsq mr k$xski xl i v$xs$egl mi zi $e$ksep>$i wxef pm
wl $e$jem
v$er h$wxer hevh${ e} $sj$q i ewyvm
r k$i | i gyxsv$t i vjsvq er gi
72Xl i $VSW$[ svph$6465$t vi wi r xexm
sr $ger $f i $vi { exgl i h$sr $] X m
j$hi wmvi h© 2023 Apex.AI, Inc. | All rights reserved.
Apex.AI, Apex.OS, Apex.Grace, Apex.Ida, Apex.Alan, Apex.Ray are trademarks of Apex.AI.

https://ros-realtime.github.io/reference-system/main/5Motivation

Extending Reference System

1. After ROS World 2021, Apex.AI took the Reference Systemand brought it in house
2. Integrated it into a nightly test pipeline running on a Linux RT
Raspberry Pi 4B with fixed CPU frequencies and isolated cores
3. By having this report available nightly, our developers could
immediately see key performance improvements orregressions that took place, and address them as needed
4. We also added another Reference System implementation totest new features of the Apex.Grace Executor© 2023 Apex.AI, Inc. | All rights reserved.
Apex.AI, Apex.OS, Apex.Grace, Apex.Ida, Apex.Alan, Apex.Ray are trademarks of Apex.AI.

6

Execution management
configuration
Kno b s to turn a nd KP Is to tra c k© 2023 Apex.AI, Inc. | All rights reserved.
Apex.AI, Apex.OS, Apex.Grace, Apex.Ida, Apex.Alan, Apex.Ray are trademarks of Apex.AI.

We will isolate specificvariables so that we
may evaluate theireffect on the system.
The KPIs are valuable
for application
developers, architects,
and system integrators
to better reason about
the desired deploymentarchitecture.

Gsr jmkyvexmsr

Execution management configuration

A handful of configuration knobs are turned between experiment runs:
••
•
•
•

Message size
Message publishing frequency
QoS configuration
Processing time (budget) per node
Operating mode of the executor

These metrics are measured andused to understand the systembehavior:•••••

CPU utilization
Memory utilization
Latency of message delivery
Dropped messages
Jitter of cyclic events© 2023 Apex.AI, Inc. | All rights reserved.
Apex.AI, Apex.OS, Apex.Grace, Apex.Ida, Apex.Alan, Apex.Ray are trademarks of Apex.AI.

8

Configuration

Operating modes for the executor

Overview of the ROS 2
Executor modes thatexist with the Iron
distro.

• Multi-threaded• Prioritized• Setting the thread priority of the hot pat h (the processing pipeline we care about)processes to the highest and set the rest of the threads to lower priorities• Single-threaded• Static single-threaded• Optimizes the runtime costs by scanning the structure of a node to determine
subscriptions, timers, service servers, action servers, etc that the node has• It performs this scan only once when the node is added, while the other two
executors regularly scan for such changes© 2023 Apex.AI, Inc. | All rights reserved.
Apex.AI, Apex.OS, Apex.Grace, Apex.Ida, Apex.Alan, Apex.Ray are trademarks of Apex.AI.

9Saving time in automotive
development
Us ing the Re fe re nc e Sys te m tod e te c t re g re s s io ns a nd im p ro ves ta b ility within Ap e x.Gra c e© 2023 Apex.AI, Inc. | All rights reserved.
Apex.AI, Apex.OS, Apex.Grace, Apex.Ida, Apex.Alan, Apex.Ray are trademarks of Apex.AI.

Experiments

Using Reference System at Apex.AI

We use the Reference System as part of our CI system for different types of regression testing:• Topology• Ensure specific and complex processing chains remain within desired latency bounds• Generate empirical evidence that shows the benefits of one solution over another, bothfor internal needs as well as in discussions with customers• Execution management• Continuous comparisons to the ROS 2 executor• We care about the Rolling branch• But any distro can be tested• Perfect companion to the (non-functional) Performance Testing framework• gitlab.com/ApexAI/performance_test© 2023 Apex.AI, Inc. | All rights reserved.
Apex.AI, Apex.OS, Apex.Grace, Apex.Ida, Apex.Alan, Apex.Ray are trademarks of Apex.AI.

11

Experiments

Let's look at the numbers

į La te nc y0$sv$xl i $xm
q i $mx$xeoi w$jsv$e$q i wweki $xs$ks$xl vsykl $xl i $Ī l sx$t exl īį M
r $xl mw$gewi $m
xw$gl swi r $xs$f i $jvsq $xl i $pm
hev$r shi w$xs$xl i $sf ni gx$gsppm
wm
sr $i wxm
q exsv0$wm
q ypexmr k$e$vi ep${ svph$wgi r evm
sį J itte r sj$e$xm
q i 1xvm
kki vi h$r shi
į Jsv$i | eq t pi 0$e$r shi $wi x$xs$f i $xvm
kki vi h$i zi v} $544q w$q e} $f i $xvm
kki vi h$wpm
kl xp} $f i jsvi $sv$ejxi v$544q w$l ew$ksr i $f }
į Me m o ry a nd CP U us a g e
į Xl i $q i q sv} $er h$GTY$yweki $$ger $zev} $hi t i r hm
r k$sr $xl i $i | i gyxsv$ywi hį Qsvi $q i q sv} $er h$GTY$yweki $m
w$r sx$r i gi wwevmp} $e$f eh$xl mr k0$wm
r gi $mx$q mkl x$nywx$q i er $xl i $i | i gyxsv$mw$t vsgi wwmr k$
q svi $hexe© 2023 Apex.AI, Inc. | All rights reserved.
Apex.AI, Apex.OS, Apex.Grace, Apex.Ida, Apex.Alan, Apex.Ray are trademarks of Apex.AI.

12

RMW: Cyclone DDS

Experiments

Latency — Humble

RMW: Fast DDS

Latency variance of the different executor modes in ROS 2 Humble to Rolling with two RMW layers© 2023 Apex.AI, Inc. | All rights reserved.
Apex.AI, Apex.OS, Apex.Grace, Apex.Ida, Apex.Alan, Apex.Ray are trademarks of Apex.AI.

13

RMW: Cyclone DDS

Experiments

Latency — Iron

RMW: Fast DDS

Latency variance of the different executor modes in ROS 2 Humble to Rolling with two RMW layers© 2023 Apex.AI, Inc. | All rights reserved.
Apex.AI, Apex.OS, Apex.Grace, Apex.Ida, Apex.Alan, Apex.Ray are trademarks of Apex.AI.

14

RMW: Cyclone DDS

Experiments

Latency — Rolling

RMW: Fast DDS

La te nc y va ria nc e o f the d iffe re nt e xe c uto r m o d e s in ROS 2 Hum b le to Ro lling with two RMW la ye rs© 2023 Apex.AI, Inc. | All rights reserved.
Apex.AI, Apex.OS, Apex.Grace, Apex.Ida, Apex.Alan, Apex.Ray are trademarks of Apex.AI.

15

RMW: Cyclone DDS

Experiments

Jitter — Humble

VQ[ >$Jewx$HHW

Execution jitter of the different executor modes in ROS 2 Humble to Rolling with two RMW layers© 2023 Apex.AI, Inc. | All rights reserved.
Apex.AI, Apex.OS, Apex.Grace, Apex.Ida, Apex.Alan, Apex.Ray are trademarks of Apex.AI.

16

I | t i vm
q i r xw

Jitter — Iron

VQ[ >$G} gpsr i $HHW

RMW: Fast DDS

Execution jitter of the different executor modes in ROS 2 Humble to Rolling with two RMW layers© 2023 Apex.AI, Inc. | All rights reserved.
Apex.AI, Apex.OS, Apex.Grace, Apex.Ida, Apex.Alan, Apex.Ray are trademarks of Apex.AI.

17

RMW: Cyclone DDS

Experiments

J itte r ╟ Ro lling

RMW: Fast DDS

Execution jitter of the different executor modes in ROS 2 Humble to Rolling with two RMW layers© 2023 Apex.AI, Inc. | All rights reserved.
Apex.AI, Apex.OS, Apex.Grace, Apex.Ida, Apex.Alan, Apex.Ray are trademarks of Apex.AI.

18

RMW: Cyclone DDSExperiments

CPU — Humble

RMW: Fast DDS

CPU consumption of the different executor modes for ROS 2 Humble to Rolling with two RMW layers
ç $6467$Et i | 2EM0$M
r g2$Å$Epp$vm
kl xw$vi wi vzi h2$$
Et i | 2EM0$Et i | 2SW0$Et i | 2Kvegi 0$Et i | 2M
he0$Et i | 2Eper 0$Et i | 2Ve} $evi $xvehi q evow$sj$Et i | 2EM
219

Experiments

CPU — Iron

RMW: Cyclone DDS

RMW: Fast DDS

CPU consumption of the different executor modes for ROS 2 Humble to Rolling with two RMW layers© 2023 Apex.AI, Inc. | All rights reserved.
Apex.AI, Apex.OS, Apex.Grace, Apex.Ida, Apex.Alan, Apex.Ray are trademarks of Apex.AI.

20

RMW: Cyclone DDS

Experiments

CPU — Rolling

RMW: Fast DDS

CPU consumption of the different executor modes for ROS 2 Humble to Rolling with two RMW layers© 2023 Apex.AI, Inc. | All rights reserved.
Apex.AI, Apex.OS, Apex.Grace, Apex.Ida, Apex.Alan, Apex.Ray are trademarks of Apex.AI.

21

In closing

Prepare your questions!

© 2023 Apex.AI, Inc. | All rights reserved.
Apex.AI, Apex.OS, Apex.Grace, Apex.Ida, Apex.Alan, Apex.Ray are trademarks of Apex.AI.

Use Reference System as part ofyour prototyping workflow
į© 2023 Apex.AI, Inc. | All rights reserved.
Apex.AI, Apex.OS, Apex.Grace, Apex.Ida, Apex.Alan, Apex.Ray are trademarks of Apex.AI.

Thank you!

evan.flynn@apex.ailyle.johnson@apex.ai© 2023 Apex.AI, Inc. | All rights reserved.
Apex.AI, Apex.OS, Apex.Grace, Apex.Ida, Apex.Alan, Apex.Ray are trademarks of Apex.AI.