SW-Architektur und –Integration für ROSbasierte Produkte
Dr.-Ing. Ingo Lütkebohle, BoschFür die ROSCon Deutschland, 23. November 2023

Agenda§ Intro§ Open Source Management§ Practices for teams§ Scaling§ Diagnostics§ Testing

Bosch Research | 2023-11-06© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

Dr.-Ing. Ingo Lütkebohle

2005 – 20132010 & 20132014-

From system integration to dependability
From HRI to embedded systems
3

Bosch Research | 2023-11-06© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

Bosch Research Focus Areas

4

Autonomous Intelligent
Driving (AID)

Chemical Energy
Converters (CEC)

Electrified Mobility and
Systems (EMY)

Healthcare Solutions(HCS)

IoT @ Life
(IOT)

Production Systems (PRS)

Sensing, perception,
prediction, planning
Systems & infrastructure
for L2-L4

Hydrogen electrolysis
PEM fuel cell for EVs
Solid Oxide Fuel Cell

Electric drives
Power electronics
Integrated electrified
products

Point-of-care lab
diagnostics
Liquid biopsy
Next gen sequencing

Enabling AIoT for mobile,
residential, tools and multidomain applications

Production technologies
AI in production
Internet of production

Robotic Systems(ROB)

Sustainability(SST)

Smart Sensors &HW Systems (SSY)

Artificial Intelligence
Methods (AIM)

Information andCommunication Tech (ICT)

Modeling, Simulation,
Optimization (MSO)

Industrialization of
AI & SW (BIS)

Consumer robots
Industrial service robots
Industrial robot arms

Sustainability big picture
Climate change mitigation
Circular economy
Carbon dioxide removal

MEMS
Quantum sensors
HMI technology
Smart systems
Embedded AI hardware

AI data loop enablers
Natural language
processing
Computer vision
AI method incubator

Software & systems
engineering
Distributed infrastructureIntelligent functions &services

Sustainable engineering
Virtual product design
Virtual validation
Use phase monitoring

Providingmature AI & SW
artefacts

Bosch Research | 2023-11-06© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

Bosch’s 12 Year Journey with ROS

Bosch participatesin PR2 beta program

2010

2011

Founding memberof ROS-I Europe

2012

2013

2014

Bosch sponsors
development of ROS 2

2015

2016

Founding memberof ROS TSC

2017

2018

EU project
micro-ROS

2019

2020

EU ITP
MROS

2021

EU project
CONVINCE

2022

Adapted fromhttp://design.ros2.org

Unconfigured

Tools and basic algorithms from PR2 beta program

Tracepoints

SUCCESS

success

configureFailureConfiguring

Cleaning Up

SUCCESS

Inactive

Clean upsuccess

activate
failureActivating

Deactivating

success

ActiveShutdown

Shutting Down

success

Finalized

rosbag2

pcg_gazebo

deactivate

Error Processing

failureNode lifecycle

Client library for C

ros2_control

System ModesUUV simulator

Diagnostics

Middleware adapter
rviz2

fmi_adapter

Refined Executor

From a small research team to hundreds of developers using ROS
5

Bosch Research | 2023-11-06© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

Bosch Research | 2023-11-06© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

7Open Source Management

What to Open Source and what not§ Common strategy for embedded companies§ Our limits- ROS 2 framework is not unique- Nav 2 framework is not unique- Motion planning: Largely unique- Perception algorithms: Largely unique(on top of OpenCV etc)
- Deliberation (e.g., behavior trees): Unique- A lot of optimization is unique
Image Source: OSADL
https://www.osadl.org/fileadmin/dam/pictures/paperless/Whitepaper-Free-and-Open-Source-Software-FOSS-ENG-Preview.png

Bosch Research | 2023-11-06© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

Steering of Our ROS Strategy and Infrastructure
Consumerrobotics

Industrial
robotics

Commercial
vehicles &off-road

Inner source

Bosch
Robotics
Middleware
TSC

Open source
Professional
service robotics
9

Bosch Research | 2023-11-06© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

Working with OSS§ Expectation management
Proof ofConcept- What can others reasonably expect§ Intellectual property- Implicit patent licenses§ License compliance

Use- Compatibility, copyleft- Ensuring licenses are correct
Contribution

à Training & Tooling§ Maintenance- Disclaiming liability is not always possible- Maintenance comes with responsibilities

Maintenance10à We prefer contribution

Bosch Research | 2023-11-06© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

Practices for team alignment

From individual to team
Essentials for effective team-work§ Shared tools§ Shared understanding§ Shared practices

Bosch Research | 2023-11-06© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

Shared Tools
ROS!
ros2_control

Image Source: https://www.ros.org/blog/ecosystem/

Bosch Research | 2023-11-06© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

Shared Tools
DevOps Tooling (and a lot of custom tooling)

14

Bosch Research | 2023-11-06© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

Product Qualities

16

Bosch Research | 2023-11-06© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

Making Qualities Tangible and Assessable§ Quality Scenarios „What do we need in our software to be sucessfull?“§ Architecture fitness functions„ Any mechanism that performs an objective integrity assessment of some architecture characteristic or
combination of architecture characteristics.” (“Evolutionary Architecture”, Ford et al, 2017)§ Examples given later

17

Bosch Research | 2023-11-06© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

From individual to team
Quality Levels: Agreement on practices§ ROS 2 added „Package Quality Categories“- REP 2004

§ Level 1: Production§ Level 4: Demos, tutorials experiments§ Level 5: Baseline§ We adopted and added to this à

18Bosch Research | 2023-11-06© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

From individual to team
Adopt DevOps Practices§ Getting rid of „it works for me“- Automated devel environment setup- Automated deployment- Configuration management§ Enabling refactoring- Automatic quality control

Image Source: https://productcoalition.com/12-top-devops-best-practices-for-a-successful-transition-in-2023-b73b54014d0d

19Bosch Research | 2023-11-06© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

20

ScalingOverarching phases for novel products§ Exploration: Before deciding to do it- Something like 10% time

Exploration§ Prototyping: Small team builds a core
- Can be stopped at any time

§ Series: Ramp-up to full product setup- Can still be stopped, but less likely

Prototyping

Series21

Bosch Research | 2023-11-06© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

Functions à Components§ Architecture must partially be ahead- Essential aspects present in common frameworks§ Separation of Concerns- Modularize for testability- Separate function and component- Factor out execution, bringup, monitoring

22

Bosch Research | 2023-11-06© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

Robot Software Architecture: Within device
Function

23

Bosch Research | 2023-11-06© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

SupportThe Integration Problem§ Software Integration as well as SW/HW Integration havelong been recognized as major challenges§ Major causes- Mismatched interfaces- Implicit assumptions / bad documentation- Bad software quality in components- “silo-thinking”- Lack of competences§ Main strategies:
1. Reduce complexity
2. Start small, grow judiciously and consciously(iteration…)
3. Use supporting frameworks

24

Bosch Research | 2023-11-06© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

Recurring issue categories in our projects§ Distributed systems- Lack of synchronization- Variability- Lossy networks- Hardware changes not communicated- In DDS: Badly connected remote subscribers
pulling down local communication- Hardware different from simulation§ Interfaces- Misuse of message fields- Static indexes for named arrays- Wrong reference frames- Time-sync causing jumps§ Complex system descriptions- Cf. complex, hierarchical launch issues mentioned
on Monday
25§ Hardware§ Quality- Bad timestamping in drivers- Lack of lifecycle in drivers- Drivers using binary-only SDKs

§ Dependency management- Lack of version pinning- Difficulty in upstreaming- Sheer number of dependencies

Bosch Research | 2023-11-06© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

Process Overview
Quality characteristics change
Exploration
Exploration

Prototype
Prototype

SeriesExploration

Changeabili
Changeability
ChangeabilitytyFunction
Function
Function
55
5
44
4

Reuse
Reuse
Reuse

333
222

Adaptability
Adaptability

Reliability
Reliability
Reliability

111
000

Modularity
Modularity
Modularity

Efficiency
Efficiency
Efficiency

SafetySafetySafetySecurity
Security
Security
Operability
Operability
Operability

26

Bosch Research | 2023-11-06© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

Simple bringup helps: 2-layer Launch§ Many ROS systems use a deep launch-fileinclusion hierarchy- This causes a great deal of duplication for passing
arguments§ Much content in launch files is also for handlingarguments§ 2-layer Launch- Bottom layer: Nodes for one subsystem- Top layer: Only includes from bottom layer- All arguments in single YAML file

- Unecessary complexity

System Launch
Subsystem 1 nodes

27

Bosch Research | 2023-11-06© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

System Config
Subsystem 2 nodes

Digitisation in robotics§ Robotics is now, more than ever, aneco-system of eco-systems- Machine learning-based functions- Advanced, model-based control- Monitoring and analytics- Edge/Cloud-based functions- Novel programming languages(e.g., Rust)- DevOps-oriented delivery

28

Bosch Research | 2023-11-06© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

Local Bag Storage à Data Management Platforms

Image Source: https://www.roboto.ai/

Image Source:
https://foxglove.dev/
29

Bosch Research | 2023-11-06© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

30

Diagnostics

From prototype to productGetting it working every time
§ Requirements- System Engineering & Integration- Handling diverse situations§ How do we get there?
- Explicit requirements, architecture, design- Establish a useful feedback loop (testing, diagnostics, metrics, real-world recordings, …)§ What is “useful”?
- Results are relevant for delivering a better product- Problems are quick to diagnose§ How do we diagnose quickly?
- Break it down into pieces- Look at both pieces and whole

Bosch Research | 2023-11-06© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

Watch out for sensor timestamping
Proper timestamp
Acquisition

Processing

Time::now()
Transmission

Kernel read

SchedulingTransmission Delay

US wake up

Scheduling Delay§ time::now à easily ~10-100ms error!
§ Transmission delay correction can be used when sensor data has no timestamp§ Scheduling delay without RT – easily 10ms
§ Use real-time scheduling!
32

Bosch Research | 2023-11-06© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

TimeMonitor for determinism

33

Bosch Research | 2023-11-06© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

Helpful tools§ Diagnostics§ Compare different versions- TopicDiagnostic à Message age/frequency§ Standard monitoring tools- Resource usage (e.g., perf, flamegraphs)- Crashes§ ros2_tracing- Callback durations- Callback sequences- Queue latencies- IO latencies- Scheduling latencies- Many more things…

34

Bosch Research | 2023-11-06© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

35

TestingAfter: „Agile Testing“, Crispin & Gregory, 2008

Process Overview
Agile Test Quadrants

36

Bosch Research | 2023-11-06© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

Testing
Test Levels
Level

Tools

Entry Criteria

Exit Criteria

Coverage Criteria

Unit-testing

Gtest, pytest, etc.

Every commit

Success

Mutation testing

Interface tests

e.g., launch_test

Every commit

Success

Architecture match

Integration tests

Launch_test,
gazebo, etc.

On pull request

Endurance testing
success

Architecture match

SW/HW integration
tests

Xray, etc.

On HAL change

Endurance successcustomPerformance tests

Various

WeeklyKPIs okay

By KPI

37

Bosch Research | 2023-11-06© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

Testing
How to come up with good tests?
§ Basic software engineering tradecraft- Some standards, such as ISO 29119§ Driven by coverage criteria§ Property-based testing§ Regression testing (if there was a bug…)- Often also based on recorded data

§ Experience38§ Generally, major source of concern and in direneed of improvement§ Areas in need of improvement- Mocking- Behavior interface tests- Speeding up tests- Test reporting/visualization- Integration of more formal tools

Bosch Research | 2023-11-06© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

Testing
Random data collection gets you only so far§ Scenario has 9866 combinations§ Correctly chosen, 21 tests can cover§ Random choice à ~5 times more effort!

Source: “Leveraging Combinatorial Testing for Safety-Critical Computer Vision
Datasets”, Gladisch et al, CVPR WS 2020

39

Bosch Research | 2023-11-06© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

ROS 2 Architecture Working Groups§ Primarily, at present- Middleware working group- Client library working group§ Future: Production Working Group

40

Bosch Research | 2023-11-06© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.

Conclusions§ Shared tools, understanding and practices are essential for sucessful product development§ Architecture methods contribute to this on several levels- Quality alignment- Scaling- Test selection§ Common architectural building blocks in ROS 2 contribute to- Shared interfaces- Diagnostics and Data Analysis- Simulation and testing

41

Bosch Research | 2023-11-06© Robert Bosch GmbH 2023. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.