Chain-Aware ROS Evaluation Tool (CARET)
Atsushi Hasegawa (Research Institute of Systems Planning, Inc.)
Keita Miura (EMB IV, Inc.)
This project is being conducted in cooperation with the University of Tokyo and TIER IV.

Background | ROS 2 Features● ROS 2 provides:○ Pub/Sub communication○ Callback scheduler⇒ ROS 2 realizes loosely-coupled, highly interoperable systems● Appropriate designing allow for a variety of systems.
Loosely coupledLow overhead(a)Many hosts(b) Single process

Example of implementation for parallel processing of nodes2Background | Response Time● Response time is important for safety-critical systems, but difﬁcult to measure.
● Cooperation of many nodes is important for system to behave as desired.

∝ Response time3

Use cases for Evaluation● Division of labor for performance evaluation and analysis.
○ Measure performance with simulation or ﬁeld testing.
○ Analyze from multiple perspectives with various teams.

Any issues?

What causes
issues?

4

Introduction to CARET●Our objective:○ Evaluating response time
○ Involving various teams in tackling performance issues
even for a large-scale ROS 2 system like Autoware.

●To achieve these objectives,
we developed Chain-Aware ROS Evaluation Tool (CARET).
For example, CARET answers:○○“Autoware response time was X ms.”
“QoS history of /cmd_vel topic was X.”

https://github.com/tier4/CARET
Apache License 2.0

5Features●Lightweight○ Add extra LTTng trace points via hook
○ Record only necessary information●Flexibility○ Provide Python-API for developers to analyze issues●

Intuitively evaluable visualization○ Support several types of intuitive ﬁgures

6Measurement Targets●CARET measures:○Callback latency○Communication latency○
Node latency○
Path latency

Major trace points and latency

7

Measurement Targets●

CARET measures:○
Callback latency○Communication latency○
Node latency○
Path latency

Major trace points and latency

8

Measurement Targets●

CARET measures:○
Callback latency○Communication latency○
Node latency○
Path latency

Major trace points and latency

9

Visualization Concept
CARET supports coarse to ﬁne granularity visualization APIs.

Any issues?

Is QoS history depthappropriate?

Were callbacks
executed instantly?

response time
● min: 100 ms● ave: 150 ms● max: 300 ms
Response timeCoarseMessage flow
Granularity

Callback scheduling
Fine

10

Key Idea｜ Message ﬂow“Which input message did the output message use?”1.
2.

Developers define node path to evaluate.
CARET draws message dependencies between I/O nodes.

1 Hzndst

coseﬁr2 Hz

However, latency in message flow is not suitable for response time evaluation.

11

Key Idea｜ Response time
"When does the system respond to events?"
1.
2.

CARET uses each first flow that reflects input message.
CARET considers worst case as well.

stndcoseﬁr

12

Diagrams and Figures
DiagramFigures by CARET
Visualize Message Flow

Visualize Response Timein Histogram13

Evaluation Steps with CARET
Evaluation steps:
1. Recording
2. Conﬁguration
3. Visualization

RecordingConfiguration

Trace Data

Architecture
File

Visualization

Figure14

Evaluation Steps｜Recording
Recording●Configuration

Generate trace data○

Trace data is historical data andconsists of information at trace points.
(timestamp, trace point’s type, message address, etc)

Visualization

RecordingTrace Dataros2 caret record(command to recordinformation)

launch target
applications
timestamp01
2…

typecallback_start
publishcallback_end
…

args
…
…
…
…

15

Evaluation Steps｜Conﬁguration
Recording●Configuration

Create an “Architecture ﬁle” which deﬁnes followings○ Target paths○ Target application structure
Visualization

Architecture File●●
●Target path
○ [node_name, topic_name, node_name, …]Executor Information○ TypeNode Information○ Callbacks Information○ Node Latency Definition

Configuration

ArchitectureFile.yaml16

Evaluation Steps｜Visualization
Recording●Configuration

Visualize the measurement results○○

CARET provides visualization APIs.
Developers can check the results with Jupyter-notebook.
Visualization

Visualization[ 1 ]:

Trace Data

[ 2 ]:#!/usr/bin/env python3
from caret_analyze.plot import Plot
… # Processing trace data and architecture
Plot.create_path_histogram(target_path)

Histogram

Plot.create_message_flow(target_path)
Message FlowArchitectureFile[ 3 ]:

Plot.create_callback_sched(target_path)
Callback Schedule

17

Measurement of an Actual SystemAutoware● An open source ROS based
autonomous driving system

Target path :● From Sensing to Control● Including major modules /nodes for autonomous
driving

Autonomous driving
configuration diagram

Sensors

AutowareSensing

Perception

Node

NodeLocalization

PlanningControl

NodeNode

Node18

Measurement Results of Autoware
Target path Histogram

Each module HistogramLong Tail.

Long Tail.

NodeHistogramLong Tail.

Sensing

Localization

Planning

Control・・・

Sensing → Localization → Planning → Control

CoarseGranularity

Fine19Future Work
Provide feedback on the perspectives of the autoware evaluation.
Expand measurement coverage○ Path containing /tf topic○ DDS layer○ System call
○ Multiple host
2. Mitigate limits and constraints○ Support complex nodes
3. Propose to integrate CARET trace points into each ofﬁcial package
1.

20Discussion and comments are welcome !

https://github.com/tier4/CARETThis presentation is based on results obtained from a project,
JPNP16007, subsidized by the New Energy and Industrial Technology Development Organization (NEDO).
T. Kuboichi, A. Hasegawa, B. Peng, K. Miura, K. Funaoka, S. Kato and T. Azumi, “CARET: Chain-Aware ROS 2
Evaluation Tool,” in Proc. of IEEE/ECU, Nov 2022 (in press)

21

Appendix

22

Examples of Complex Nodes
Possible to measureSub
node latency
deﬁnication

receivemsg

The node receiving dataand publishing the data.

sub_cb = [](&msg){
msg_ = func(msg);
publish(msg_)}

The node receiving somedata and publishing one
data.

sub_cb = [](&msg){
queue.push(msg)msg_ = avg(queue);publish(msg_)}

The node using the data at a
speciﬁc time.

sub_cb = [](&msg){
msg_ = lookup(msg.stamp);
publish(msg_)}

Pubpublish msgtimeImpossible to measure
SubPubSub
Pub23

Overhead
Measurement Method
msg_->data = "Hello World";
start_time = now();
pub_->publish(msg_);
end_time = now();
exe_time = end_time - start_time;

additional 4~7 tracepoints

24Output Figures with CARET
CoarseSystemLevel

Visualization Granularity

Fine

Scalable

TargetGranularity

Callback
Level

25

Trace Points●

Add trace points to record information.
○○○○

Time (e.g. callback start)
Implementation (e.g. node name)
Configuration (e.g. QoS)
Message identifier (address, message stamp)※ Skip recording by hook if nodes or topics are unnecessary.

Tracepoints added layers

Hooking with LD_PRELOAD

26

Deﬁnition of target paths● Deﬁne a path to evaluate from a node graph.
target path●

●Search to select a path○ search(A, E) # CARET API
Enumerate the paths to be written in
one stroke from node A to node E.
returns:■ [A,C,E]■ [A,D,E]
Deﬁne manually○ It is possible to specify the pathmanually.

27

Scheduling Visualization●●Callback scheduling affects node latencies.
Scheduling visualization decompose node latencies
into callback latencies and scheduling latencies.

28