Evolving Messages Over Time - REP-2011
William Woodall <william@openrobotics .org>
Brandon Ong <brandon@openrobotics .org>
You Liang Tan <tan_you_liang@hotmail.com>
Kyle Marcey <kyle.marcey@apex.ai>
October 2022

Overview● Conceptual Overview of REP-2011● New Features Needed in ROS 2● Dive into Run-Time Interface Reflection● Future Work and Known Issues

REP-2011

Motivation for REP-2011
Reasons for this REP:
● It is natural for types to evolve over time
○ In your projects○ And in ROS 2 its elf● We need tools to detect when this happens● We need tools to help trans ition between vers ions

The Problem
In ROS 2 today:● If you try to change a type unevenly acros s yours ys tem:

○ Some middlewares may allow communication,depending on the change○ Warnings /errors when types are incompatible vary
○ Limited features expos ed in ROS 2 to help youevolve types in a backward/forward way

The Problem - Example# Temperature.msg
uint64 timestamp
int32 temperature

What if we want
to change this?

The Problem - Example
What if we want
to change this?

# Temperature.msg
uint64 timestamp
int32 temperature
Add a new, butredundant, optional
field?

❓

# Temperature.msg
uint64 timestamp
int32 temperature
optional float64 temperature_float❓

Just change
the type
directly?

# Temperature.msg
uint64 timestamp
float64 temperature

Note: you can’t actually do this right now.

The Problem - Example
What if we want
to change this?

# Temperature.msg
uint64 timestamp
int32 temperature
You can do either, if
the middleware
supports it.

✅

# Temperature.msg
uint64 timestamp
int32 temperature
optional float64 temperature_float✅

We’ll focus on
this option, forthe REP.

# Temperature.msg
uint64 timestamp
float64 temperature

Note: you can’t actually do this right now.

The Proposed Solution● REP-2011:○ https://github.com/ros -infrastructure/rep/pull/358● REP-2011 aims to help users:○ Know when messages have changed○ Convert between versions on demand○ Write code to convert between versions● It will do so by depending on the ability to:○ Interact with types using only their description● This REP does not try to:○ Expose “advanced” serialization features like
optional fields, extensible types, or inheritance○ Prevent these “advanced” features from working

What will this look like in practice?
% ros2 topic echo /scan sensor_msgs/msg/LaserScan[WARN] [1666081526.522630000] [ros2_bag]: Publisher '[gid...]' on topic '/scan' isusing a version of 'sensor_msgs/msg/LaserScan' ('abc123') that does not match theversion used locally ('def456').

% ros2 interface transfer_functions info sensor_msgs/msg/LaserScan abc123 def456
Conversion available with transfer functions:- [abc123 -> cba321]:- pkg: sensor_msgs_migration- description: new field added to describe ...
- [cba321 -> def456]:- pkg: sensor_msgs_migration- description: changed the type of field ...

What will this look like in practice?

What will this look like in practice?
% ros2 interface convert_topic_types \--from /scan \--to /updated/scan \# --component-container <container name>% ros2 topic echo /updated/scan sensor_msgs/msg/LaserScan...

What will this look like in practice?
● Ways to set up conversions:○ as a stand-alone node
○ as a node component○ syntactic sugar in a launch file
● Benefits of this approach:○ keeps QoS and queuing in middleware○ easy to observe from tools (e.g. rqt_graph)● Downsides of this approach:○ requires extra topics and hops through pub/sub○ requires transfer functions to exist

Needed Underlying Technical Changes● TypeDescription.msg○ Description of Other Message/Service/etc.

● Type Version Hashing and Enforcement○ Generation and Access via ROS Graph APIs

● Type Description Distribution○ Accessing definition of types remotely● Run-Time Interface Reflection○ Interacting with types using only theTypeDescription, i.e. reading, writing, sending, andreceiving

Run-Time Interface Reflection

The Context
Normally for ROS:
Message files → Compile-time Generated Code and Headers● E.g. String.msg → std_msgs::msg::String(std_msgs/msg/string.hpp)

But what if…
● You don’t have the message headers● But you obtain the message description at runtime○ E.g. From a bagfile, published over a topic, etc.

Run-time Interface Reflection“At run -time, given a byte buffer and its description…
Can we access its members?”
6d 65 74 68 79 6c 44
72 61 67 64 6e ...
{‘fields’: [(0, ‘string’)]}“raw_byte_ser”

●
●
●
●

List fields
Get and set fields
Construct new message instances…

Pub-Sub (At run-time)
Using interface reflection , and a message
description , dynamically create at run-time…
Mus t be pars ed(Dynamic Type )Mus t allow get/s et(Dynamic Data )Pub-Sub (At run-time)Pub
1. Parse description to create a dynamic type
2. Use dynamic type to create dynamic data
3. Publish dynamic data
Sub
1. Receive dynamic data
2. Parse description to create a dynamic type
3. Use dynamic type to access dynamic data

How will run-time interface reflection be
implemented?

Reflection for Different Technologies
For most technologies, we can just create a wrapper
Technology

Dynamic TypeDynamic Data

FastRTPS (C++)

DynamicType

DynamicData

RTI Connext (C)DDS_TypeCode

DDS_DynamicData

Protobuf (C++)

FileDescriptorProto

DynamicMessage

…

…

…

For middlewares that don’t have s tructured mes s ages ,we can jus t piggyback off any s erialization library (e.g. Fas tCDR)
The type des cription helps retain type information!

Run-time Interface Reflection LibraryThis functionality should be a standalone C librarys eparately from RMW.

that can be us ed

The run -time interface reflection library should abstract away
serialization! (By piggybacking off a middleware or wrapping a s erializationlib!)

This library s hould be able to s upport non-DDS, non-XTypes
libs ! (e.g. Protobuf/Zenoh)Demo
We made a prototype to check for feas ibility and refine the interfaces✨It WORKS with Fas tDDS pub-s ub!! ✨(And there’s a protobuf dynamic example too!!)Future Work

Future Work
● More prototypes (e.g. Connext)● Create the interfaces to abstract away getters andsetters● Type description distribution● Plug it all into rcl/rmw!

Known Issues● Some bugs need to be fixed in middlewares
related to run-time interface reflection● Some middlewares lack the necessary interfaces
right now
● Some conceptual discrepancies in the typedescription message, e.g. bounded sequences ofbounded strings

Ques tions
This presentation: https ://bit.ly/3dFitlq
The REP PR: https ://tinyurl.com/rep-2011-pr

Pres entation Slides

Appendix

Pub-Sub

Demo: It works ! (Fas tDDS Pub-Sub)

Pub-s ub Demo Codetinyurl.com/fastdds-ets-pubsub

DemoWe made a prototype to check for feas ibility and refine the interfaces✨

It WORKS with Fas tDDS pub-s ub!! ✨(And there’s a protobuf dynamic example too!!)Demo: Type Description

FastDDS Prototype: Type Description
The type_des cription_t s truct allows us to
iterate through the fields and obtainneces s ary information to cons truct thetype.

FastDDS Prototype: ETSThe EvolvingTypeSupport (ETS) is a
C interface to be filled by any downs tream
implementations !

FastDDS Prototype: ETS

With the type_des cription_t s truct, we caniterate through the fields and call theneces s ary methods to create the type!

Demo: It works! (FastDDS Pub)
You can us e the s ame interface on thes ubs cription s ide!

This is grabbed at runtime!

Demo: It als o works for protobuf (no protoc)!
In this cas e the DynamicType comesfrom a runtime-generated .proto file!
We us e the DynamicMessage interfaces
from protobuf to cons truct the mes s age
from the generated .proto file contents

The demo repo includes theproto file generator library! (protogen)tinyurl.com/protobuf-dyn-ser