ROS 2 Network Monitoring
ROSCon 2022

Speaker:● Raúl Sánchez-Mateos, Project Manager @ eProsima

raul@eprosima.com

October 20th, 2022

Agenda
01

03

Motivation● Introduction to thestatistics toolkit● Available data in thestatistics toolkit

Fast DDS
Statistics Backend● Library overview● Demo: Exporting statistics
data to Prometheus andvisualizing with Grafana

02

04

Fast DDS
Statistics Module● Introduction to theStatistics Module● How to enable Fast DDS
Statistics on ROS 2

ROS 2 Monitor● GUI overview● Demo: Visualize
demo_nodes_cpp statistics● Demo: Network bandwidth
at discovery phase2

Motivation
Use cases of the Fast DDS statistics toolkit

Real-time
network statusmonitoring

Measureendpoints
discovery time

Show the data supplied by a
network monitor.

Computate globally the total timeof the discovery phase per eachpair of endpoints.

3

Motivation
ROS 2 user action items

Minimizecommunications
latency

Maximizepublication'sthroughput

Minimize
packet loss

The network monitor will report
the communications latency.

Monitor the throughput so that therate of packets/bytes sent per
second can be maximized.

Monitor packet loss to reduce thenumber of lost/dropped packets byadjusting the publication settings.

4

Reported data
Available statistics data in the Fast DDS Statistics toolkit

1

Fast DDS Latency

2Throughput
Subscription’s throughput
Publication’s throughput

3

Re-sent DATA packets

4

Discovery time5

Sub-messages sent by each DDS entity and usedin the communication:
HEARTBEAT (DataWriter)
ACK (DataReader)
NACK (DataReader)
GAP (DataWriter)
DATA (DataWriter)

6

Meta-traffic packets transmitted by each DDS
entity:Discovery meta-traffic
PDP packets
EDP packets
Reliability meta-traffic
HEARBEATs
ACKNACKs

5

Fast DDS Statistics Toolkit
Fast DDS statistics toolkit components within the ROS 2 stack

Application
Custom User App

rclcpprclpy

rcljavarcl
rmw

rmw_fastrtps

FAST DDS
6

Fast DDS Statistics Toolkit
Fast DDS statistics toolkit components within the ROS 2 stack

Application
Custom User App

rclcpprclpy

rcljavarcl
rmw

rmw_fastrtps

1Statistics Module
FAST DDS
7

Fast DDS Statistics Toolkit
Fast DDS statistics toolkit components within the ROS 2 stack

Application
Custom User App2rclcpp

rclpy

rcljavarcl
rmw

rmw_fastrtps

1

Statistics Module
FAST DDS
8

Fast DDS Statistics Toolkit
Fast DDS statistics toolkit components within the ROS 2 stack

Application

3

Custom User App

2rclcpprclpy

rcljavarcl
rmw

rmw_fastrtps

1

Statistics Module
FAST DDS
9

Fast DDS
Statistics Module
A Fast DDS extension thatenables the recollection of data
concerning
communication.

theDDSFast DDS Statistics Toolkit
Fast DDS statistics toolkit components within the ROS 2 stack

Application
Custom User App

rclcpprclpy

rcljava
rclrmw

rmw_fastrtps

1Statistics Module
FAST DDS
11

Fast DDS Statistics Module
Key concepts and operation of Fast DDS Statistics Module● Extension of DDS Layer● Fast DDS collects the data describing its operation● Internal DomainParticipant (ROS 2 context)● One builtin Publisher for each data type(latency, throughput, data lost, etc.)● One can subscribe to this topics directly to retrieve the raw data
_fastdds_statistics_history2history_latency_fastdds_statistics_subscription_throughput_fastdds_statistics_heartbeat_count_fastdds_statistics_gap_count_fastdds_statistics_sample_datas_fastdds_statistics_discovered_entity_fastdds_statistics_network_latency_fastdds_statistics_rtps_sent_fastdds_statistics_acknack_count_fastdds_statistics_data_count_fastdds_statistics_pdp_packets_fastdds_statistics_physical_data_fastdds_statistics_publication_throughput_fastdds_statistics_rtps_lost_fastdds_statistics_nackfrag_count_fastdds_statistics_resent_datas_fastdds_statistics_edp_packets

12Fast DDS Statistics Module
Main concerns about Fast DDS Statistics Module in ROS 2● Available in:○ ROS 2 Humble (Fast DDS v2.6.2)○ ROS 2 Galactic (Fast DDS v2.3.5)● Disabled by default● Enabled at compilation

13

How to enable Statistics
Steps to enable Fast DDS Statistics Module on ROS 2 Humble

1🔾🔾Install ROS 2 Humble2🔾🔾

Build Fast DDS from sources with FASTDDS_STATISTICS=ON$ colcon build --cmake-args -DFASTDDS_STATISTICS=ON3🔾🔾

Source the ROS 2 Humble environment$ source /opt/ros/humble/setup.bash

4🔾🔾Source the built Fast DDS environment (overlay)$ source <path/to/fastdds_ws>/install/setup.bash

At this point the Statistics Module is compiled but not enabled!
5🔾🔾

Enable by XML configuration file or setting an environment variable$ export FASTDDS_STATISTICS="HISTORY_LATENCY_TOPIC;NETWORK_LATENCY_TOPIC;PUBLICATION_THROUGHPUT_TOPIC;SUBSCRIPTION_THROUGHPUT_TOPIC;RTPS_SENT_TOPIC;RTPS_LOST_TOPIC;HEARTBEAT_COUNT_TOPIC;ACKNACK_COUNT_TOPIC;NACKFRAG_COUNT_TOPIC;GAP_COUNT_TOPIC;DATA_COUNT_TOPIC;RESENT_DATAS_TOPIC;SAMPLE_DATAS_TOPIC;PDP_PACKETS_TOPIC;EDP_PACKETS_TOPIC;DISCOVERY_TOPIC;PHYSICAL_DATA_TOPIC"

14Fast DDSStatistics Backend
C++ library to collect data fromthe Fast DDS Statistics Module,andgeneratestatisticalinformation to be used byapplications.

Fast DDS Statistics Toolkit
Fast DDS statistics toolkit components within the ROS 2 stack

Application
Custom User App2rclcpp

rclpy

rcljavarcl
rmw

rmw_fastrtps
Statistics Module
FAST DDS
16

Main features
Main features of the Fast DDS Statistics Backend

Records the DDS entities (DomainParticipant,DataReader / DataWriter, Locator, Domain and Topic)

In-memory database

C++ library to access entities
discovered and its statistics issued
by Fast DDSDeploy DataReaders subscribed tostatistics topics (Statistics Module)

and Physical entities (Host, User, Process)discovered in a Domain

Compute the statistics (mean, std. dev,median, max, min, sum) on the stored data

Full documentation and examples
available on ReadTheDocs.

17

Statistics, Prometheus & Grafana
Exporting statistics from ROS 2 using the Fast DDS Statistics Backend to Prometheus and visualizing with Grafana

18

Statistics, Prometheus & Grafana
Exporting statistics from ROS 2 using the Fast DDS Statistics Backend to Prometheus and visualizing with Grafana

19ROS 2Monitor
GraphicalUserInterface

tomonitor a ROS 2 network inreal-time,withintrospection

methods and statistical datameasure capabilities.

Fast DDS Statistics Toolkit
Fast DDS statistics toolkit components within the ROS 2 stack

Application

3

Custom User App

rclcpprclpy

rcljavarcl
rmw

rmw_fastrtps
Statistics Module
FAST DDS
21

ROS 2 Monitor
Overview of ROS 2 Monitor graphical user interface

22Normal operation
Normal operation of ROS 2 Monitor

23

Main features
Listing DDS and physical entities

1
List of all DDS andphysical entities
discovered24

Main features
Initializing a monitor

2Initialize a monitor ina DDS Domain or in a
Discovery Servernetwork

25

Main features
Compatible with ROS 2 Discovery Server networks

3

Initialize Monitor
in a Discovery
Server network26

Main features
Real-time statistics visualization

Visualize in real-time
the main statistics ofeach entity427

Main features
Live configurable charts for every reported data
Create5interactive
real-time charts

28

Main features
Exporting recorded data in CSV format

6

Export all
recorded datain CSV format

29

Custom features for ROS 2
Monitoring meta-traffic ROS 2 nodes and topics

130

Custom features for ROS 2
Hiding meta-traffic ROS 2 nodes and topics

231Custom features for ROS 2
Monitoring actual user data ROS 2 nodes and topics

332

Talker & Listener network statistics
Visualize talker & listener demo_nodes_cpp network statistics with Fast DDS Monitor

33

Discovery traffic analysis
Network traffic comparative between default ROS 2 nodes discovery and ROS 2 Discovery Server34www.eProsima.com

Linkedin.com/company/eProsima

Twitter.com/EProsima