188* ROS bugs later:
Where do we go from here?
Christopher Timperley
ROSCon 2019, Macau

1

●
●
●
●

program repair
fuzzing
testing
program transformation2
https://www.microsoft.com/en-us/research/uploads/prod/2018/11/airsim_blog_1400x788-800x550.jpg
https://upload.wikimedia.org/wikipedia/commons/thumb/6/6a/Quadcopter_camera_drone_in_flight.jpg/1920px-Quadcopter_camera_drone_in_flight.jpg

An unlikely beginning...

3

An unlikely beginning...

4

5

6

7

8

What kinds of bugs do ROS developers encounter?

Understanding ROS bugs
https://i1.sndcdn.com/artworks-000469545585-rkzn0o-t500x500.jpg
https://thomasgriffin.io/wp-content/uploads/2014/11/toolbox.png

Tools, Tips, and Techniques
9Meet the ROBUST team!
Program
RepairStaticAnalysis

SoftwareQuality

Robotics

10

ROBUST:ROS Bug Study
11

How wecollected
our bugs
Nature vector created by macrovector - www.freepik.com
https://en.wikipedia.org/wiki/Pitfall!#/media/File:Pitfall!_Coverart.pn
https://i-love-png.com/images/science-lab-png-picture.pngg
https://en.wikipedia.org/wiki/Pitfall!#/media/File:Pitfall!_Coverart.png

Commondeveloper
pitfalls

Improveyour ROS
code today

Building
better ROS
code fortomorrow 12

We studied 188 266 bugs across 8 ROS packages
Subject# Bugs

MAVROS

57

Kobuki40

Universal Robot

25

Motoman22

TurtleBot

12

Care-O-Bot

11

geometry2

36

Confidential

10

13
https://www.turtlebot.com/assets/images/turtlebot_2_lg.png

We analysed 1790 GitHub issues and pull requests
Subject# Bugs# Issues

MAVROS

57

325

Kobuki

40

623

Universal Robot

25

158

Motoman

22

78

TurtleBot

12170

Care-O-Bot

11

182

geometry2

42

254

Confidential

10

N/A

14

We forensically described each bug
id: f01d952
title: No Image Coming From Camera
description: >A missing runtime dependency crashed the image processing node,causing no images to be received from the camera.
classification: Missing Dependency (no CWE)
keywords: ['camera', 'eigen', 'dependencies', 'runtime']
system: turtlebot
severity: error
bug:phase: runtime-operation
specificity: ROS-specificarchitectural-location: application-specific code
application: nulltask: visionsubsystem: driver
package: ros-perception/image_pipeline/depth_image_proc
languages:- C++
detected-by: runtime crash
reported-by: contributor
issue: https://github.com/turtlebot/turtlebot/issues/142
time-reported: 2014-06-12 (10:04)
reproducibility: always
trace: |[ INFO] [1402564815.530736554]: /camera/rgb/camera_info -> /camera/rgb/camera_info[ERROR] [1402564815.727176562]: Failed to load nodelet [/camera/depth_metric_rect] of type[depth_image_proc/convert_metric]: Failed to load library/opt/ros/indigo/lib//libdepth_image_proc.so.
Make sure that you are calling the PLUGINLIB_EXPORT_CLASS macro in the library code, and that...

15

We built a Docker image for each bug
id: f01d952
title: No Image Coming From Camera
description: >A missing runtime dependency crashed the image processing node,causing no images to be received from the camera.
classification: Missing Dependency (no CWE)
keywords: ['camera', 'eigen', 'dependencies', 'runtime']
system: turtlebot
severity: error
bug:phase: runtime-operation
specificity: ROS-specificarchitectural-location: application-specific code
application: nulltask: visionsubsystem: driver
package: ros-perception/image_pipeline/depth_image_proc
languages:- C++
detected-by: runtime crash
reported-by: contributor
issue: https://github.com/turtlebot/turtlebot/issues/142
time-reported: 2014-06-12 (10:04)
reproducibility: always
trace: |[ INFO] [1402564815.530736554]: /camera/rgb/camera_info -> /camera/rgb/camera_info[ERROR] [1402564815.727176562]: Failed to load nodelet [/camera/depth_metric_rect] of type[depth_image_proc/convert_metric]: Failed to load library/opt/ros/indigo/lib//libdepth_image_proc.so.
Make sure that you are calling the PLUGINLIB_EXPORT_CLASS macro in the library code, and that...

https://www.silicon.co.uk/wp-content/uploads/2019/02/Docker-logo-012-440x330.png

16

We used a time machine to build historically-accurate Docker images- tar:
local-name: actionlib
uri: https://github.com/ros-gbp/actionlib-...
version: actionlib-release-release-indigo-actionlib-1.11.3-0- tar:
local-name: angles
uri: https://github.com/ros-gbp/geometry_angles...
version: geometry_angles_utils-release-release-indigo-angles-1.9.9-0- tar:
local-name: catkin
uri: https://github.com/ros-gbp/catkin-release/archive/...
version: catkin-release-release-indigo-catkin-0.6.14-0...

rosinstall_generator_time_machine

deps.rosinstall

Try it out for yourself: https://bit.ly/2NnkmSr
17
https://www.silicon.co.uk/wp-content/uploads/2019/02/Docker-logo-012-440x330.png
https://ya-webdesign.com/image/clip-movis-the-time-machine/991524.html

What kinds of bugs do ROS
developers encounter?

18
https://www.mysecuritysign.com/blog/wp-content/uploads/2017/05/booby-trap.jpg

Incorrect type casts and conversions
Missing or incorrect frame conversions
Missing build-time dependencies
Missing run-time dependencies
Using the wrong robot model
Unintentional use of defaults
Incorrect use of namespacing
Incorrect mathematical operations
Lack of synchronisation leads to race condition
Bad name remappings
Incorrect constant values
Hardcoded values lead to silent errors
Encoding errors
Circular dependencies
Buffer overflow
API changes
Ignoring stop instruction
Inconsistent naming
Missing input validation
Missing variable initialisation
Incorrect data parsing
Python run-time errors
Use of obsolete functions
Incorrect type casts and conversions
Missing or incorrect frame conversions

Circular dependencies
Buffer overflow
API changes
Ignoring stop instruction
Inconsistent naming
Missing input validation
Missing variable initialisation
Incorrect data parsing
Python run-time errors
Function call with incorrect arguments
Uncaught exceptions
Infinite loops
Off-by-one errors
Use of global rather than local namespace
Use of obsolete functions
Incorrect type casts and conversions
Missing or incorrect frame conversions
Missing build-time dependencies
Missing run-time dependencies
Using the wrong robot model
Unintentional use of defaults
Incorrect use of namespacing
Incorrect mathematical operations
Lack of synchronisation leads to race condition
Bad name remappings
Incorrect constant values
Hardcoded values lead to silent errors
Encoding errors

Off-by-one errors
Use of global rather than local namespace
Use of obsolete functions
Incorrect type casts and conversions
Missing or incorrect frame conversions
Missing build-time dependencies
Missing run-time dependencies
Using the wrong robot model
Unintentional use of defaults
Incorrect use of namespacing
Incorrect mathematical operations
Lack of synchronisation leads to race condition
Bad name remappings
Incorrect constant values
Hardcoded values lead to silent errors
Encoding errors
Incorrect type casts and conversions
Missing or incorrect frame conversions
Missing build-time dependencies
Missing run-time dependencies
Using the wrong robot model
Unintentional use of defaults
Incorrect use of namespacing
Incorrect mathematical operations
Off-by-one errors
Use of global rather than local namespace
Use of obsolete functions
Incorrect type casts and conversions

More than a few...

19

1. Missing system build dependencies!
<?xml version="1.0"?><package format="2"><name>libmavconn</name><version>0.9.0</version><description>
MAVLink communication library.
This library provide unified connection
handling classes and URL to connection
object mapper.
...
</description>…

+<buildtool_depend>catkin</buildtool_depend><depend>boost</depend><depend>mavlink</depend><depend>libconsole-bridge-dev</depend><test_depend>gtest</test_depend>...
</package>id: dc1327f
title: Missing mavlink dependency
system: mavros
description: >Versions 0.9 and 0.8.3 of MAVROS fail to build due to a missing"mavlink" dependency.

-- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~-- ~~ traversing 1 packages in topological order:-- ~~ - libmavconn-- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~-- +++ processing catkin package: 'libmavconn'-- ==> add_subdirectory(repo-under-test/libmavconn)
CMake Error at repo-under-test/libmavconn/CMakeLists.txt:10 (find_package):
By not providing "Findconsole_bridge.cmake" in CMAKE_MODULE_PATH thisproject has asked CMake to find a package configuration file provided by"console_bridge", but CMake did not find one.
Could not find a package configuration file provided by "console_bridge"
with any of the following names:
console_bridgeConfig.cmakeconsole_bridge-config.cmake
Add the installation prefix of "console_bridge" to CMAKE_PREFIX_PATH or set"console_bridge_DIR" to a directory containing one of the above files. If"console_bridge" provides a separate development package or SDK, be sure it
has been installed.

-- Configuring incomplete, errors occurred!
See also "/ros_ws/build/CMakeFiles/CMakeOutput.log".
See also "/ros_ws/build/CMakeFiles/CMakeError.log".
Invoking "cmake" failed

20Complication: Have to be specified in multiple locations<?xml version="1.0"?><package format="2"><name>libmavconn</name><version>0.9.0</version><description>
MAVLink communication library.
This library provide unified connection
handling classes and URL to connection
object mapper.
...
</description>…

cmake_minimum_required(VERSION 2.8.3)
project(libmavconn)
find_package(catkin REQUIRED)+ find_package(console_bridge REQUIRED)
find_package(Boost REQUIRED COMPONENTS system)...
catkin_package(
INCLUDE_DIRS include
LIBRARIES mavconn+
DEPENDS Boost console_bridge mavlink
CFG_EXTRAS libmavconn-extras.cmake)include_directories(
include${CMAKE_CURRENT_BINARY_DIR}/catkin_generated/include${Boost_INCLUDE_DIRS}${mavlink_INCLUDE_DIRS}+${console_bridge_INCLUDE_DIRS})...
target_link_libraries(mavconn${Boost_LIBRARIES}+${console_bridge_LIBRARIES})+<buildtool_depend>catkin</buildtool_depend><depend>boost</depend><depend>mavlink</depend><depend>libconsole-bridge-dev</depend><test_depend>gtest</test_depend>...
</package>21

2. Missing runtime dependencies!
●
●
●
●
●

Shared objects / DLLs
Python modules
ROS nodes
Configuration files...

id: f01d952
title: No Image Coming From Camera
system: turtlebot
description: >A missing runtime dependency crashed the image processing node,causing no images to be received from the camera.
...

[ INFO] [1402564815.530736554]: /camera/rgb/camera_info ->/camera/rgb/camera_info[ERROR] [1402564815.727176562]: Failed to load nodelet[/camera/depth_metric_rect] of type[depth_image_proc/convert_metric]: Failed to load library/opt/ros/indigo/lib//libdepth_image_proc.so.
Make sure that you are calling the PLUGINLIB_EXPORT_CLASS macro in thelibrary code, and thatnames are consistent between this macro and your XML. Error string: Could notload library (Poco exception = libeigen_conversions.so: cannot open sharedobject file: No such file or directory)[FATAL] [1402564815.727410623]: Service call failed!

22

3. Dangerous defaults# mavros/mavros/scripts/mavcmd
def do_long(args):
try:
ret = command.long(broadcast=args.broadcast,command=args.command,
confirmation=int(args.confirmation),param1=args.param1,
param2=args.param2,
param3=args.param3,
param4=args.param4,
param5=args.param5,
param6=args.param6,+

id: c6791f0
title: Missing parameter in ROS service callsystem: mavros
description: >The "do_long" function in the "mavcmd" script accepts an object
that describes a MAVLink command. Command objects are suppliedwith seven arguments, which may be used or ignored by the command
handler depending on the type of command. The "do_long" function
dispatches this command object to the "mavros.command.long"
function. which forwards the command to MAVLink. Rather than
accepting positional arguments, the "mavros.command.long" accepts
all of its arguments as keywords. Missing keywords are assumed tobe irrelevant and substituted by a nominal placeholder value.
The buggy implementation of this function fails to pass along theseventh argument during the call to "mavros.command.long",
causing incorrect information to be forwarded to MAVLink. Thisbug will only manifest for commands which utilise all sevenarguments.

param7=args.param7)
except rospy.ServiceException as ex:
fault(ex)_check_ret(args, ret)

Also see: #e1a8005; #ff581a0

23

4. String identifiers: Silent but dangerous<!-Auto-docking + keyop configuration for working with the default kobuki
launcher (minimal.launch).
--><launch><node pkg="nodelet"
type="nodelet"
name="cmd_vel_mux"
args="load cmd_vel_mux/CmdVelMuxNodeletmobile_base_nodelet_manager"><param name="yaml_cfg_file"value="$(find kobuki_auto_docking)/param/cmd_vel_mux.yaml"/><remap from="cmd_vel_mux/mux_cmd_vel”+<remap from="cmd_vel_mux/output”
to="mobile_base/commands/velocity"/></node>...
</launch>

id: 35682ec
system: kobuki
title: Wrong topic remapping makes robot ignore velocity commands
description: >ROS allows resource names to be remapped. Topics, in particular,
are resources that can be remapped. When topic "A" is remapped to"B" and a node tries to publish/subscribe on "A", it will
(automatically) publish/subscribe to "B" instead.
Remappings are commonly defined in launch files. Launch files are
special XML files used by the tool `roslaunch` to initialize
several nodes, parameters and remappings at once.
In this particular bug, a node (cmd_vel_mux) publishes velocity
messages that are received by the driver node (mobile_base). A
launch file defined a remapping from "cmd_vel_mux/mux_cmd_vel" to"mobile_base/commands/velocity", but the cmd_vel_mux node is, infact, publishing on "cmd_vel_mux/output". Since the node ispublishing on a different topic, the remapping has no effect, andso the two expected topic names are not connected. The end result
is that the published messages never reach their destination,i.e., the robot does not move at all.
This issue is very specific to ROS, as it is located and solvedin launch.xml files, but the issue of a runtime configuration ofa distributed system is general, in principle.
...

Also see 43705f7, b18f559, fbe70c7, 263650d

245. Namespace confusion — the defaults strike back!
class VisualizationPlugin : public MavRosPlugin {public:VisualizationPlugin() : viz_nh("~visualization"), ... { };+/visualization/visualization/fixed_frame_id/visualization/fixed_frame_id

void initialize(...) {...
viz_nh.param<std::string>("visualization/fixed_frame_id",viz_nh.param<std::string>("fixed_frame_id",
fixed_frame_id, "local_origin");...
}

Also see #1518978, #9ffffca, #a482f82

id: 84264f0
title: Incorrect use of namespacing in ROS parameter names
description: >All ROS parameter interactions within"mavros_extras/src/plugins/visualization.cpp" incorrectly prefix
the name of parameters with a "visualization" namespace. This
namespace is implicit, and so the fully qualified parameter that
result are incorrect. e.g., "visualization/fixed_frame_id"
becomes "/visualization/visualization/fixed_frame_id" when fully
qualified.
classification: "Incorrect use of ROS namespacing (no CWE)"...

25

How can I build
better software
today?
26
https://img5.goodfon.com/wallpaper/nbig/f/ca/tools-toolbox-floor.jpg

Make your life easier with continuous integration●●●●

Identify unspecified build-time dependencies!
Use smoke tests to find missing run-time dependencies
Use test automation to identify regressions
Incorporate other tools into CI pipeline...

27

Use linters to statically catch errors before run-time

●
●
●
●

roslint
catkin_lint
ament_lint (ROS2)...

28

Use sanitizers to catch runtime errors earlier
GCC and Clang offer sanitizers that perform various safety checks on your code:
●
●
●
●
●
●
●
●

Buffer overflows
Memory leaks
Initialization order bugs
Data races
Deadlocks
Uninitialized memory
Undefined behaviour...

$ catkin_make \-DCMAKE_CXX_FLAGS=”-fsanitize=address”

https://github.com/google/sanitizers

29Use static type checking in Python (PEP 484)

Roughly one in twelve
bugs was a type error!

from typing import Iterator
def fib(n: int) -> Iterator[int]:a, b = 0, 1
while a < n:yield aa, b = b, a + b

def is_palindrome(s):# type: (str) -> bool
return s == s[::-1]

30

What's next?

31

Mithra: Oracle learning for simulation-based testing●●●

Operates on logged data
(e.g., ROS bag files)
Uses multivariate time series
clustering to identify intended systembehaviours
Uses anomaly detection to identify
erroneous executions

32

HAROS: A static analysis framework for ROS apps
●
●
●

●

Understands package andworkspace structure
Unifies reports from various tools(e.g., metrics, linters)
Able to extract the ROS Graphfrom code (C++ as of now;
Python in the next release)
Produces interactive reports

https://github.com/git-afsantos/haros/

33

Shout out to Phriky-Units
Detects physical unit inconsistencies for C++ without the need for annotations.

https://github.com/unl-nimbus-lab/phriky-units34

$ rosinstall_generator_time_machine

ROBUST:ROS Bug Study

What kinds of bugs do ROS
developers encounter?

How can I build
better software
today?

What next?

github.com/robust-rosin/robust
35