Building dependable
products with ROS
Dr.-Ing Ingo Lütkebohle
Bosch Research
For ROSCon JP, September 26th, 2023

Dr.-Ing. Ingo Lütkebohle

2005 – 20132010 & 20132014-

From system integration to dependability
From HRI to embedded systems
2

Ingo Lütkebohle | 2023-08-29© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

About BoschOur business sectors

Mobility Solutions

Industrial Technology

Energy and Building
Technology

Ingo Lütkebohle | 2023-08-29© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

Consumer Goods

Bosch Research Focus Areas

Autonomous Intelligent
Driving (AID)

Chemical Energy
Converters (CEC)

Electrified Mobility and
Systems (EMY)

Healthcare Solutions(HCS)

IoT @ Life
(IOT)

Production Systems(PRS)

Sensing, perception,
prediction, planning
Systems & infrastructure
for L2-L4

Hydrogen electrolysis
PEM fuel cell for EVs
Solid Oxide Fuel Cell

Electric drives
Power electronics
Integrated electrified
products

Point-of-care lab
diagnostics
Liquid biopsy
Next gen sequencing

Enabling AIoT for mobile,
residential, tools andmulti-domain applications

Production technologies
AI in production
Internet of production

Robotic Systems(ROB)

Sustainability(SST)

Smart Sensors &HW Systems (SSY)

Artificial Intelligence
Methods (AIM)

Information andCommunication Tech (ICT)

Consumer robots
Industrial service robots
Industrial robot arms

Sustainability big picture
Climate change mitigation
Circular economy
Carbon dioxide removal

MEMS
Quantum sensors
HMI technology
Smart systems
Embedded AI hardware

AI data loop enablers
Natural language
processing
Computer vision
AI method incubator

Software & systems
engineering
Distributed infrastructureIntelligent functions &services

4

Ingo Lütkebohle | 2023-08-29© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

Modeling, Simulation, Industrialization
Optimization (MSO) of AI & SW (BIS)
Sustainable engineering
Virtual product design
Virtual validation
Use phase monitoring

Providingmature AI &SW artefacts

501

Robotics at BoschIngo Lütkebohle | 2023-08-29© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

Bosch’s 12 Year Journey with ROS

Bosch participatesin PR2 beta program

2010

2011

Founding memberof ROS-I Europe

2012

2013

2014

Bosch sponsors
development of ROS 2

2015

2016

Founding memberof ROS TSC

2017

2018

Adapted fromhttp://design.ros2.org

Tools and basic algorithms from PR2 beta program

Unconfigured

Tracepoints

SUCCESS

success

configureFailureConfiguring

Cleaning Up

SUCCESS

Inactive

Clean upsuccess

Error Processing

activate
failureActivating

Deactivating

success

ActiveShutdown

Shutting Down

success

Finalized

EU project
micro-ROS

2019

rosbag2

2020

EU ITP
MROS

2021

EU project
CONVINCE

2022

pcg_gazebo

deactivate

failureNode lifecycle

Client library for C

ros2_control

System ModesUUV simulator

Middleware adapter
rviz2

fmi_adapter

Refined Executor

From a small research team to hundreds of developers using ROS
7

Ingo Lütkebohle | 2023-08-29© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

Diagnostics

Steering of Our ROS Strategy and Infrastructure
Consumerrobotics

Industrial
robotics

Commercial
vehicles &off-road

Bosch
Robotics
Middleware
TSC

Open source
Professional
service robotics

8

Inner source

Ingo Lütkebohle | 2023-08-29© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

902

Why ROS?

Academic Frameworks before ROS
Year

Framework(s)Citations

1994

TCA

~500

1997

DAMN

~700

1998

SmartSoft

<20

1999

ISR

<10

2000

OSCAR

<20

2001

OROCOS
MCA

>800
~100

2002

RoboMote

>500

2003

Player/StageCARMEN
ROCI

>2000
~300
~50

2006

YARP
MOOS
ORCA

>800
~250
~200

2007

STAIR

~150

2009

ROS

>110002010

LCM
Fawkes~400~80

 Industrial frameworks largely robot-specific Academic Open Source Frameworks
 These have many similarities on the plumbing
level

Ingo Lütkebohle | 2023-08-29© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

ROS Success Factors
Which of these is most important?

Image Source: https://www.ros.org/blog/ecosystem/

Ingo Lütkebohle | 2023-08-29© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

Building an ecosystem

12

Ingo Lütkebohle | 2023-08-29© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

Virtuous cycle
ROS Core•API reviews•Tool plugins•Examples

Capabilities•HW Drivers•Perception•Planning•Manipulation•Datasets

Community Growth•Conferences•Trainings•Papers

13

Ingo Lütkebohle | 2023-08-29© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

14

03

ROS in the real world

15„The Office Marathon“, 2010
E. Marder-Eppstein, E. Berger, T. Foote, B. Gerkey and K. Konolige, "The Office Marathon: Robustnavigation in an indoor office environment," 2010 IEEE International Conference on Robotics andAutomation, Anchorage, AK, USA, 2010, pp. 300-307, doi: 10.1109/ROBOT.2010.5509725.

 26.2 miles, 30 hours, no collision, one graze.

Can we use this in production?

We tried it in 2015
 And we got really bad
performance in narrowsituations

Ingo Lütkebohle | 2023-08-29© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

Assumptions vs. Reality
 PR2 in „office marathon“

 Our robot

 Consequences− 1kHz odometry− 50Hz odometry− Pose uncertainty larger− 30Hz LIDAR− 12.5 Hz Safety LIDAR− 8 Core Xeon CPU− 2 core Atom CPU− Inflation radius needs tolarger

− 2.5cm costmap resolution− 10 cm costmap resolution− 20 Hz control rate

− 10 Hz control rate

− less space to manouver Slower speed
 Delayed reactions

On re-use, check assumptions…
17

Ingo Lütkebohle | 2023-08-29© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

But: A clash of worlds
LIDAR-based safety:
Speed limit zone
Stop zone

18

Ingo Lütkebohle | 2023-08-29© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

The challenge of determinism
 Navigation amongst humans− Using ROS move_base

 Problem: Robot kept grazing obstacles thatwere still at a distance in the costmap

Image Source:
K. Adam, A. Butting, R. Heim, O. Kautz, J. Pfeiffer, B. Rumpe, A. Wortmann
Modeling Robotics Tasks for Better Separation of Concerns, Platform-Independence, and Reuse.
In: Shaker Verlag, ISBN 978-3-8440-5319-7. Aachener Informatik-Berichte, Software Engineering, Band 28. December 2017.

20

Ingo Lütkebohle | 2023-08-29© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

System checks for navigation determinism
 Mismatch between reality and model?
 Check sensor rate Check sensor timing
 Check model updates

 Mismatch between control and execution
 Check plant model Check control dead time

21

Ingo Lütkebohle | 2023-08-29© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

Most common sensor timestamping issues
 Basic sensor processing pipeline:− Gray is sensor-side, blue is Linux-side

Acquisition Processing

Transmission

Kernel read

Scheduling Many drivers used „time::now()“− Too late!
 Error up to sensor period (tens to hundreds of milliseconds)

 Some subtract offset for „transmission delay“ or „acquisition delay“− But… no real-time scheduling− And no priorities in ROS
Error easily 10ms or more due to process scheduling

22

Ingo Lütkebohle | 2023-08-29© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

US wake up

TimeStill trouble with people Dynamic obstacles, such as persons, stillpose issues− Robot started avoidance motion too late

 We examined sensor data age and found this:Last-update age of costmap at time of control computation

23

Ingo Lütkebohle | 2023-08-29© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

Processing in Costmap
Receive Scan

Spin Thread

Receive Pose

waitForTF

Update Costmap
Write

Costmap
Read

Control Rate

Compute Control CommandControl Thread Time Definitions:− 𝑇𝑇𝑆𝑆𝑛𝑛 LIDAR scan n is complete− 𝑇𝑇𝑈𝑈𝑛𝑛 costmap is updated with scan n− 𝑇𝑇𝐶𝐶𝑘𝑘 local planner starts computing control k „Sensor Data Age“: 𝑇𝑇𝐴𝐴𝑘𝑘 = 𝑇𝑇𝑆𝑆𝑚𝑚 − 𝑇𝑇𝐶𝐶𝑘𝑘 , 𝑚𝑚 = max𝑚𝑚=1..𝑁𝑁 𝑇𝑇𝑈𝑈𝑚𝑚 < 𝑇𝑇𝐶𝐶𝑘𝑘
24

Costmap Thread

Ingo Lütkebohle | 2023-08-29© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

Publish cmd_vel

Tracepoint

Sensor Data Age Plot

25

Ingo Lütkebohle | 2023-08-29© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

Add synchronization
 Add notification so that controller runs after costmap update See ROSCon 2017 talk „Determinism in ROS – or when things break sometimes and how to fix it“ formore details− https://vimeo.com/236186712− https://roscon.ros.org/2017/presentations/ROSCon%202017%20Determinism%20in%20ROS.pdf

35

Ingo Lütkebohle | 2023-08-29© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

27

04

Micro-ROS

Micro-ROS
 Microcontrollers have been separate
 Goals
Wire Sensor− ROS concepts on the MCU

Nav

Safety MCU− Seamless interoperability with ROS 2− Wide hardware/RTOS support

 And: Better determinism…

Motor
Control MCU

Comm MCU

Wire Sensor

Image source: Bosch PowerTools GmbH, All rights reserved
Ingo Lütkebohle | 2023-08-29© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

Development of the micro-ROS project2017

 Ideation
 OFERA EU project29

 Architecture One board
 ROS Embedded WG

2018

2019

 Open-source demo
with Turtlebot
 Build tooling for ROS
developers
 Start of rclc

2020

 More boards Triggered executor

Ingo Lütkebohle | 2023-08-29© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

2021

2022

 Build tooling for typicalembedded developers Successful final review
of EU project OFERAMicro-ROS Architecture
Application
component…

Application
component

C++ API(rclcpp)
C API
rcl

…

Application
component

functions,rclc: Convenience
deterministic execution, …

ROS Middleware Interface (rmw)
ROS 2
Agent Ethernet,

Bluetooth,
Serial

Linux
uP
31

Micro XRCE-DDS Client
POSIX+ Additional abstractions

Zephyr, FreeRTOS, NuttX
microcontroller

Ingo Lütkebohle | 2023-08-29© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

Additional
drivers, …

FULL PORTABILITY
Any RTOS and Bare metal Library
Generator!
Any low-mid range MCU!

REFERENCE HW
Renesas RA6M5
ArduinoTypical features:Raspberry Pi Pico~ 150 KB of flash memory> 25 KB of RAM memory
General purpose input/output pins
Peripherals: GPIO, USB, Ethernet,
SPI, UART, I2C, CAN, etc

Arduino Nano RP2040 Connect

REFERENCE
RTOS
Mbed RTOS 6.8 / 6.9 / 6.10
FreeRTOS
NuttX 10.0 / 10.1

ESP-IDF v4.3 & ESP32-S2/C3
Teensy 3.2 / 3.5 / 4.1 / 4.2

Zephyr RTOS 2.4 / 2.5

OpenCR support
STM32

Azure RTOS ThreadX

Crazyflie 2.1 drone, ...
Ingo Lütkebohle | 2023-08-29© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

Check full list of supported HW & RTOS
https://micro.ros.org/docs/overview/hardware/

Build Systems
For your ROS tooling:$ git clone github.com/micro-ROS/micro_ros_setup.git$ colcon build$ ros2 run micro_ros_setup↳ create_firmware_ws.sh [RTOS] [MCU]↳ configure_firmware.sh [MY_APP] -t [TRANSPORT]↳ build_firmware.sh↳ flash_firmware.sh$ ros2 run micro_ros_setup↳ create_agent_ws.sh↳ build_agent.sh↳ micro_ros_agent [TRANSPORT]For your favorite tool:
Renesas e2 studio
STM32CubeMX/IDE
RasPi Pico SDK
ArduinoESP32 IDF
ZephyrIngo Lütkebohle | 2023-08-29© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

Triggered Executor Concept
Extended API for typical patterns• Control loops• Data fusion

IMU

Laser

@500Hz

@10Hz

@10Hz

sense

Σ

+

• Prioritized paths
Ingo Lütkebohle | 2023-08-29© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

35

05

Notes from BR2

BR2 Challenges
 Rich sensor-set− LIDAR, stereo camera rig, environmental sensors, close distance sensors

 Heterogenous compute− ARM-based CPUs, GPU, NN-accelerator

 Scalable Autonomy− Full autonomy to full tele-op from afar

36

Ingo Lütkebohle | 2023-08-29© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

Leveraging Nav2
 Used for P2P and path-following

x

x

y

y

d

Cartesian coordinates
37

Ingo Lütkebohle | 2023-08-29© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

s=0dFrenet coordinates

Image source: https://github.com/fjp/frenet Path-following uses DWB with custom critics
to achieve sufficient precision

Hierarchical Behavior Tree Approach Bosch uses BTs at least since 2015

SetBlackboardIN:

output_key

mission_stage

SetBlackboard
IN:

value

false

 Reduce complexity / improve performancethrough hierarchical BTs coupled with actions
 Image on right is full top-level BT – not socomplex, isn’t it?

value

initializing
IN/OUT:

IN/OUT:

output_key

mission_failed

SetBlackboard
IN:

value

true
IN/OUT:

output_key

mission_started_from_dock

SetBlackboard
IN:

value

false
IN/OUT:

output_key

undocking_done

SetBlackboard
IN:

value

false
IN/OUT:

output_key

online_coverage_done

Sim pleWait
IN:

w ait_duration2

ClearSlam WorldModel
IN:

IsRoom ByRoom Mission

service_name/slam_w orld_model/clear_mission_data
Root

Sequence

Sequence

PrintText

init_then_run_mission

IN:

Fallback
PrintText

print_mission_mode
IN:

text_to_print

The robot is running a room by room mission

text_to_print

The robot is running a full coverage mission

InverterForceSuccess

IsRobotStuck

TestIfRobotOnDock_autoremap+
SetBlackboard
IN:
ForceFailure Pro: Individual trees compact

value

unstucking
IN/OUT:

Fallback

output_key

mission_stage

perform_unstuck_action_if_robot_gets_stuck

PrintText
IN:

RetryUntilSuccessfulIN:text_to_print

Robot is stuck! Trying to unstuck
Sequence

num_attempts

5
Perform UnstuckAction
PrintText
ForceFailureIN:

text_to_print
SetBlackboard

Could not unstuck the robot! FAIL this mission!
IN:

value

undocking Con: Overall structure harder to see

IN/OUT:

output_key

mission_stage

PrintTextIN:

text_to_print

UndockingBlackboardCheckBool

IN:

IN:

ReactiveSequence

Sequencevalue_A

ForceSuccessreturn_on_mismatch

SUCCESS

return_on_mismatch

SUCCESSIN:

RelocalizeOnDock

mission_started_far_from_dock

BlackboardCheckBool
skip_undocking_if_already_done_it

perfom_undocking_and_save_start_pose

IN:

IN:

IN:

True

value_B_autoremap

Sequencevalue_A

relocalize_on_dock_and_undock

Undock{mission_started_from_dock}{undocking_done}_autoremap

value_B+

false

run_mission

SetBlackboard
IN:

value

true
IN/OUT:

output_key

undocking_done
PrintTextIN:

HandleStartPoseAnnotation

mission_started_from_dock

false
IN:

text_to_print

Perform online coverage

add_start_pose_to_w m_anyw ay
IN:

SetBlackboard

operation

IN:

add

value

cleaning

IsRoom ByRoom Mission

IN/OUT:

output_key

mission_stage

Sequence

Room ByRoom Coverage_autoremap

IfThenElse

BlackboardCheckBool

IN:

IN:

return_on_mismatch

SUCCESSIN:

value_A

IN:

value

IN/OUT:

do_online_coverage

online_coverage_done

SetBlackboard
IN:

IN/OUT:

value
Sequence

IN/OUT:

output_key

mission_stage

PrintText
IN:

PrintText
text_to_print

PrintTextIN:

text_to_print

Robot started on the dock, w ill now drive to the dock!
Sequence
mission_started_from_dock
GoToDock
Sw itch2_autoremap+

mission_end_case_selector
case_1PrintText

true
IN:

case_2BlackboardCheckStringIN:

mission_did_not_start_on_dock

return_on_mismatch

Alw aysFailure
should_not_happen

SUCCESSIN:

true© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

Sequence

GoToStartPose

value_A{mission_failed}

Ingo Lütkebohle | 2023-08-29

text_to_print

Robot started aw ay from dock, w ill now drive to start_pose!
variable{mission_started_from_dock}

IN:

38

IN:

false
IN:

value_B

text_to_print

Online coverage failed!

DockingIN:

output_key

mission_failed

docking

IN:

value

true

SetBlackboard

https://convince-project.eu/

output_key

value_B

false

IN:

Alw aysFailure_autoremap+IN:timeout{mission_timeout}

true
Fallback{online_coverage_done}
Sequence
undock_then_online_coverage_then_back

Perform OnlineCoverage

SetBlackboard

skip_online_coverage_if_already_done_it++Leveraging micro-ROS
IMU

Rangers

Switches

Stamped Pose, Safety State

Safety MCU
MotorsNon-Safe MPU
Time, Base Velocity

 Pro: Auto-generated messages, time-sync, transport-agnostic
 Con: RAM usage, perceived complexity

39

Ingo Lütkebohle | 2023-08-29© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

2-layer Launch Many ROS systems use a deep launch-fileinclusion hierarchy− This causes a great deal of duplication for
passing arguments

 Much content in launch files is also forhandling arguments

 2-layer Launch− Bottom layer: Nodes for one subsystem− Top layer: Only includes from bottom layer− All arguments in single YAML file

− Unecessary complexity

System Launch
Subsystem 1 nodes

40

Ingo Lütkebohle | 2023-08-29© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

System Config
Subsystem 2 nodes

DDS implementation assumptions vs Robotics needs
DDS Assumptions

Robotics Needs

 You need different QoS settings

 Topics need different QoS

 Throughput is more important than latency Throughput and latency are both important Network situations are stable Networks are ad hoc

 Users able to configure networking precisely

 Users (generally) know little about networks

 Multicast works

 Multicast causes networks to fail

 Everybody may want to talk to everybody

 Most connections are 1-1

 Capacity for plenty of discovery traffic

 Network capacity is very limited

 One socket is enough for all topics

 Participants need to isolated

 Tooling needs to be DDS specific and detailed

 Tooling needs to be integrated

41

Ingo Lütkebohle | 2023-08-29© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

42

Conclusions

The story continues in 2023…
 Bosch Engineering Group
worked with Hako on theautonomous Scrubmaster
B75, based on ROS 1

 Meanwhile, Bosch Rexroth
also offers the VDA5050
compliant „ROKITNavigator“, based on ROS 2
Navigation 2
43

Ingo Lütkebohle | 2023-08-29© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

Conclusions
 ROS and ROS 2 have been an important
foundation for our research for over a decade ROS 2 has become more versatile but also
more complex

 Journey The layered approach has enabled things
such as Micro-ROS and different executors− Tooling for development only
− Prototyping in early stages− Product fully based on ROS 2

General Know and check your assumptions!
 Become a contributor, it’s worth it

44

Ingo Lütkebohle | 2023-08-29© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

Thank you!
Questions?
(Japanese is okay, will be translated for me)

45

Ingo Lütkebohle | 2023-08-29© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.