Lessons Learned During TRADR
Search & Rescue Robotics Using ROS
M. Pecka, S. Caccamo, R. Dubé, V. Kubelka, D. Reuter

TRADR GitHub

September 21st-22nd, 2017
Vancouver, Canada

tradr-project.eu

Lessons Learned During TRADR, ROSCon 2017, 22 Sep. 2017
M. Pecka, S. Caccamo, R. Dubé , V. Kubelka, D. Reuter

Outline
1.
2.
3.
4.
5.

TRADR project intro
Our ROS tools and packages
Advanced functionalities of ourrobotsOur hardware experience
Bonus

Presentation available at bit.do/tradr

Lessons Learned During TRADR, ROSCon 2017, 22 Sep. 2017
M. Pecka, S. Caccamo, R. Dubé , V. Kubelka, D. Reuter

TRADR Project
●
●

●
●

●

UGVs and UAVs as a part ofhuman-robot rescue teams
Exploration, detection andmonitoring in a disaster area,sample pickup missions
Strong end-user orientation
Human ↔ Robot cooperation,
every team member does what
he’s better at
Robots need to work in(unfriendly) real world withoutmarkers, pretrained models
etc.

Presentation available at bit.do/tradr(EU - FP7 Framework)Lessons Learned During TRADR, ROSCon 2017, 22 Sep. 2017
M. Pecka, S. Caccamo, R. Dubé , V. Kubelka, D. Reuter

Team Members

Presentation available at bit.do/tradr

System architecture

Lessons Learned During TRADR, ROSCon 2017, 22 Sep. 2017

Presentation available at bit.do/tradr

M. Pecka, S. Caccamo, R. Dubé , V. Kubelka, D. Reuter

UGVs

UAVs

●
●
●
●

●
●
●

●

Custom-made tracked vehicles
Ca. 25 kg (55 lbs)
Wi-Fi connection (Ubiquiti Bullet, 5 GHz)
Multiple cameras, tilting lidar,
tactile/smoke/gas/radioactivity/.. sensors
No GPS (intentionally)

More models, all commercial
Live streaming of RGB and IR cameras
D-GPSLessons Learned During TRADR, ROSCon 2017, 22 Sep. 2017
M. Pecka, S. Caccamo, R. Dubé , V. Kubelka, D. Reuter

OCU (Operator Control Unit)●●

Multiple RQT plugins
Modular design for different UGV
configurations (e.g. presence of amanipulator)

Presentation available at bit.do/tradr

TDS (Tactical Display System)●●

Multiple RQT plugins
Interface between high-level reasoning
and humans

Lessons Learned During TRADR, ROSCon 2017, 22 Sep. 2017
M. Pecka, S. Caccamo, R. Dubé , V. Kubelka, D. Reuter

OCU (Operator Control Unit)●●

Multiple RQT plugins
Modular design for different UGV
configurations (e.g. presence of amanipulator)

Presentation available at bit.do/tradr

TDS (Tactical Display System)●●

Multiple RQT plugins
Interface between high-level reasoning
and humans

UGV operators

UAV operator

Team leader

ROS Tools & Packages

Lessons Learned During TRADR, ROSCon 2017, 22 Sep. 2017

Presentation available at bit.do/tradr

M. Pecka, S. Caccamo, R. Dubé , V. Kubelka, D. Reuter

Workspace Layout and Versioning●

Problem: >100 ROS packages
○

●
●
●

Still not there (do better than us!)

they need to be organized and separable

We use git, but not submodules
Ca 15 git repos
Some machines only need some repos●

Autoinstall scripts for developers were
offered, but nobody uses them○○●

We partly use rosinstall○●

Still too confusing for some developers

Free choice of build tool○●

Complicates adding non-ROS
dependencies and repo-repo relations
They’re used for monthly integration tests

catkin_make / catkin tools / catkin simple

We use our own Gitlab installation○

Has both pros and cons

Lessons Learned During TRADR, ROSCon 2017, 22 Sep. 2017
M. Pecka, S. Caccamo, R. Dubé , V. Kubelka, D. Reuter

Presentation available at bit.do/tradr

Multimaster (customizations released)●

Problem: Too much data transmitted overwireless, TCP congestion●

Nimbro_network with customizations○●

Support latched and bi-directional topics

Problems: no auto connect/disconnect,
network resilience still isn’t perfect

Lessons Learned During TRADR, ROSCon 2017, 22 Sep. 2017

Presentation available at bit.do/tradr

M. Pecka, S. Caccamo, R. Dubé , V. Kubelka, D. Reuter

Multimaster (customizations released)●

Problem: Too much data transmitted overwireless, TCP congestion●●

Nimbro_network with customizations

●

○

●

Support latched and bi-directional topics

Problems: no auto connect/disconnect,
network resilience still isn’t perfect

YAML configs and launch file generator(not released, but planned)
Easy configuration of video, dynamic
reconfigure and actions

ctu-robot.yaml

packs/axis_ptz.yaml

robot:
hostname: ctu-robot
prefix: ugv1

to_robot:udp:- name: "axis/control/look_at"- name: "axis/control/zoom/relative"

ports:
to_core:tcp: 17011
udp: 17012
service: 17013
to_robot:
tcp: 17014
udp: 17015
service: 17016
packs:- base- teleop- axis_ptz

to_core:video:- name: "axis"
dynamic_reconfigure:- name: "axis/axis"
udp:- name: "axis/camera/ptz"

Lessons Learned During TRADR, ROSCon 2017, 22 Sep. 2017
M. Pecka, S. Caccamo, R. Dubé , V. Kubelka, D. Reuter

Presentation available at bit.do/tradr

TRADR Orchestra (not released)●

Problem: Too many parts of the systems
to be launched, some only sometimes●

Visual control over a set of scripts/launchfiles that run everything needed during themissionNo “one launch file that rules them all” →it’s easy to restart parts of the system
Status reporting (running, dead)
Scripts run in separate screen sessions
Unified deployment script (rsync)

●
●
●
●

main:
{{ hostname }}:- name: UGVshell: ugv.sh- name: UGV_multishell: ugv_multi.sh {{ ugv_prefix }}- name: relay
shell: relay.sh {{ ugv_prefix }} {{ core_hostname }}- name: new_mapper
shell: new_mapper.sh {{ ugv_prefix }} {{ localize_against_map }}Lessons Learned During TRADR, ROSCon 2017, 22 Sep. 2017
M. Pecka, S. Caccamo, R. Dubé , V. Kubelka, D. Reuter

Presentation available at bit.do/tradr

TRADR Orchestra (not released)●

Problem: Too many parts of the systems
to be launched, some only sometimes●

Visual control over a set of scripts/launchfiles that run everything needed during themissionNo “one launch file that rules them all” →it’s easy to restart parts of the system
Status reporting (running, dead)
Scripts run in separate screen sessions
Unified deployment script (rsync)

●
●
●
●

main:
{{ hostname }}:- name: UGVshell: ugv.sh- name: UGV_multishell: ugv_multi.sh {{ ugv_prefix }}- name: relay
shell: relay.sh {{ ugv_prefix }} {{ core_hostname }}- name: new_mapper
shell: new_mapper.sh {{ ugv_prefix }} {{ localize_against_map }}Lessons Learned During TRADR, ROSCon 2017, 22 Sep. 2017
M. Pecka, S. Caccamo, R. Dubé , V. Kubelka, D. Reuter

Presentation available at bit.do/tradr

Dynamic URDF Model Generation (not released)●

Problem: Additional sensors/equipment
differ for each mission●

Laser filtering, camera image masking,useful for teleoperation, coarse interactive
calibration of sensor positions
Implemented using Xacro anddynamic_reconfigure
No need to restart drivers to add/remove
sensors/equipment●●<robot name=”NIFTi”><xacro:arg name="has_jaco_arm" default="0" /><xacro:if value="$(arg has_jaco_arm)"><xacro:include filename="$(find kinova_description)/
urdf/j1n6a300.xacro"/><xacro:j1n6a300 base_parent="jaco_base_helper" />….

Lessons Learned During TRADR, ROSCon 2017, 22 Sep. 2017
M. Pecka, S. Caccamo, R. Dubé , V. Kubelka, D. Reuter

Presentation available at bit.do/tradr

Tracked Robot Simulation in Gazebo (released)●

Problem: (Proper) tracked vehicle
simulation hasn’t been available inGazebo●Teaser for our IROS presentation “FastSimulation of Vehicles with Non-deformable
Tracks” (WeCT4.2)
Enables us doing Reinforcement learningwith the UGVThe simulation model is also dynamicallygenerated from Xacro template●●Lessons Learned During TRADR, ROSCon 2017, 22 Sep. 2017
M. Pecka, S. Caccamo, R. Dubé , V. Kubelka, D. Reuter

Presentation available at bit.do/tradr

camera_info_manager for Zoom Cameras (released)●

Problem: We have a zoom camera andwe need its calibration data●●

Python only (good for e.g. IP cameras)
Basic support for zoom cameras incamera_info_manager_py
Two approaches:

●

○
○

●

Calibrate at several zoom levels
A trigonometry-only solution

Integration into general ROS packages iscomplicatedLessons Learned During TRADR, ROSCon 2017, 22 Sep. 2017
M. Pecka, S. Caccamo, R. Dubé , V. Kubelka, D. Reuter

Multiple PTZ Camera RQT Widget●

Problem: Need to be controlled by 1
operator●

Automatic detection of running cameras(by getting # of publishers)
Abstraction over multiple pan-tilt-zoom
control APIs
360° panoramic view for quick orientationand visualization of field of view
Basic control over video stream andcamera properties
The thermo camera view allows settingcolor palette properties

●
●
●
●

Presentation available at bit.do/tradr

Lessons Learned During TRADR, ROSCon 2017, 22 Sep. 2017

Presentation available at bit.do/tradr

M. Pecka, S. Caccamo, R. Dubé , V. Kubelka, D. Reuter

High-level and Low-level Databases

Low-level DB●●●

MongoDB (NoSQL) +GridFS for large files
New database for eachmissionROS integration using
catkin-generated get* andput* services for various
data types

CMakeLists.txt:
add_tradrdb_msg(Image sensor_msgs Image)autogen_srv/getImage.srv:
string object_id--sensor_msgs/Image data

Semantic modelers●●

Transform low-level (raw) data
into high-level (semantic)
information○ Fire/victim detector withspatial clustering○ Actor poses○ Operator workload
Should not work with “live”
ROS streams, but with datafrom MongoDB

High-level DB●●

SparQL semantic database withontology
Used by “agents” in GOAL frameworkand tactical displays.
.
.

Lessons Learned During TRADR, ROSCon 2017, 22 Sep. 2017
M. Pecka, S. Caccamo, R. Dubé , V. Kubelka, D. Reuter

Presentation available at bit.do/tradr

Easy-to-use C++ Tensorflow (released)●

Our hacky way to easily use Tensorflow in ROS
and C++○○●●

No need to use/install Bazel - compilation is doneusing catkin
Requires C++11 support in compiler (not in Trusty
by default)(Mis)uses the pip-installed version of Tensorflow
for Python
It isn’t optimized to recent CPUs, so performance
is not good enough for training neural networks,
but sufficient enough for running simple ones

package.xml (format 2)...
<build_depend>tensorflow_ros</build_depend><exec_depend>tensorflow_ros</exec_depend>...

CMakeLists.txt
find_package(catkin REQUIRED COMPONENTS
tensorflow_ros)
add_library(tensorflow_executor
src/TensorflowGraphExecutor.cpp)
target_link_libraries(tensorflow_executor${catkin_LIBRARIES})
set_target_properties(tensorflow_executor PROPERTIESCXX_STANDARD 11 CXX_STANDARD_REQUIRED YES
CXX_EXTENSIONS NO)TRADR Functionalities

Lessons Learned During TRADR, ROSCon 2017, 22 Sep. 2017
M. Pecka, S. Caccamo, R. Dubé , V. Kubelka, D. Reuter

Presentation available at bit.do/tradr

3D Laser Mapping (released)
SLAM system for multiple robots equipped with
3D LiDARs
●
●
●
●
●

Back-end: Incremental pose-graphoptimization
ICP-based LiDAR odometry constraints
Loop-closures via SegMatch - a segment
extraction and matching approach
Framework available open-source withdemonstrations
More information at IROS SLAM 1 session
MoBT7.2

https://github.com/ethz-asl/segmatch

Lessons Learned During TRADR, ROSCon 2017, 22 Sep. 2017
M. Pecka, S. Caccamo, R. Dubé , V. Kubelka, D. Reuter

Presentation available at bit.do/tradr

Heterogeneous 3D Map Merging
Global Registration of maps from 3D ground
LiDAR and aerial reconstruction:●●●

Inputs aerial 3D reconstructions fromcamera and LiDAR sub-map from UGVKey-point descriptor-based global
registration
Registered aerial map enables continuous
localization, traversability analysis, andpath planning

Gawel, A., Dubé, R., Surmann, H., Nieto, J., Siegwart, R., & Cadena, C.. 3D Registration of Aerial and Ground Robots for Disaster Response: An
Evaluation of Features, Descriptors, and Transformation Estimation." SSRR (2017).

Lessons Learned During TRADR, ROSCon 2017, 22 Sep. 2017
M. Pecka, S. Caccamo, R. Dubé , V. Kubelka, D. Reuter

Presentation available at bit.do/tradr

3D Multi-robot Path Planning, PatrollingMore info @ https://sites.google.com/a/dis.uniroma1.it/3d-cc-patrolling/Lessons Learned During TRADR, ROSCon 2017, 22 Sep. 2017
M. Pecka, S. Caccamo, R. Dubé , V. Kubelka, D. Reuter

Presentation available at bit.do/tradr

Resilient Communication-Aware Motion Planner●●

Mobile robots require stablecommunication with the base station
during a USAR mission.
Stochastic elements in radio signal
propagation and possibility of
unpredictable events or hardware failuresoften lead to signal loss.

Disconnected robots are either abandoned or
attempt to autonomously back-trace their way
to the base station.

Lessons Learned During TRADR, ROSCon 2017, 22 Sep. 2017
M. Pecka, S. Caccamo, R. Dubé , V. Kubelka, D. Reuter

Presentation available at bit.do/tradr

Resilient Communication-Aware Motion Planner
State-of-the-art methods:●●●

Uses a priori knowledge of the network structure(e.g. access point locations)
Missing an effective self-repair strategy
Do not handle complex scenarios and rewuires
some offline processing(fingerprinting/training, etc.)

What is needed?

A system that allows to dynamicallymap the RSS distribution andautonomously moves the robot to aconnection safe location in case of
signal loss avoiding obstacles.

More information at IROS - Motion and Path Planning I session MoCT13.2

Lessons Learned During TRADR, ROSCon 2017, 22 Sep. 2017
M. Pecka, S. Caccamo, R. Dubé , V. Kubelka, D. Reuter

Presentation available at bit.do/tradr

Assisted Grasping❏

Collecting a sample during a USAR mission is a difficult task (time constraints, limited fov, delays, many
DOF to control)➔

Solution: vision aided cartesian arm control (using a SIFT based sample detector)

TRADR Hardware Experience* The following slides are not objective reviews of products. They represent our experience with mentioned products.
In most cases, we did not do comprehensive practical comparisons of available alternatives.

Lessons Learned During TRADR, ROSCon 2017, 22 Sep. 2017

Presentation available at bit.do/tradr

M. Pecka, S. Caccamo, R. Dubé , V. Kubelka, D. Reuter

USB Cameras●●

We were quite unlucky in our selection of
USB cameras for various purposes
When selecting a new USB camera, wewould:○○○●

Thoroughly check its support in LinuxkernelCheck if it has a decent ROS support
Run a series of tests on different battery
levels of the robot

●

○
○

○
○
○
○

have a lot of USB connectors directly on
the motherboard (with screws)
not use unnecessary cable extenders
provide at least 1.0 A even to USB 2 ports
have enough USB 3 ports
have a separate USB bus for every port

Problems with longer cables and cable
extenders
Problems when more cameras in one USB
hub (even with external power)
■

●

○

●

Picky about input power “shape” (onnotebook OK, on robot 0.2 FPS)
Panicks with our 4.4 Linux kernel whenconnected at boot timeIntel RealSense R200○●

Micro-Epsilon thermoIMAGER TIM 160

IDS uEye○

When designing a new robot, we would:○

Asus Xtion Pro

Doesn’t work with cable extenders

Logitech C920○

Bulky but reliableLessons Learned During TRADR, ROSCon 2017, 22 Sep. 2017

Presentation available at bit.do/tradr

M. Pecka, S. Caccamo, R. Dubé , V. Kubelka, D. Reuter

Wireless Communication●●

We have tried several wirelesscommunication options to achieve
operational radius of the robots about 200
meters in urban areas
WiFi 2.4 GHz○○●

Military grade radio-based mesh network○○●

Good real bandwidth (~100 Mbps)
Nicely surprised in a furnace made of steel
Long range, real bandwidth ~32 Mbps
Illegal if you’re not government

Custom-made 1:1 866MHz radio link○○

Range up to 500 m, bandwidth ~200 bps
For emergency in case fast link is lost

Ubiquity Bullet M2/M5
○
○

○

●

○

●

We use it on the robots
Allows using non-standard 2.4 GHz bands
and proprietary AirMAX protocol which
gives higher bandwidth
Can be fixed to one AP by MAC address

Zyxel AC1200 (USB 3 dongle)○

WiFi 5 GHz 802.11a/ac/n○○●

Not enough bandwidth for our system
2.4 spectrum usually occupied●

Good support in Linux, but latency was
higherQuite powerful despite its size

Netgear Nighthawk AC1900 (router)○○Good performance on 5 GHz 802.11ac
Has problems when 2 clients are too close

Lessons Learned During TRADR, ROSCon 2017, 22 Sep. 2017

Presentation available at bit.do/tradr

M. Pecka, S. Caccamo, R. Dubé , V. Kubelka, D. Reuter

Lidar / Depth Sensors●●●

We use them for several purposes - metricmapping (and 3D planning), virtual
bumper, reactive control and visual aid forhuman operators
Each use requires a different quality/typeof laser scans (speed, density,
completeness, coarseness of self-filtering)
We would like to remove the mechanicallytilted Sick laser, but still haven’t found acomparable non-moving alternative
○

●

●

○
○
○
○

●

●

* K. Zimmermann et al., Learning for Active 3D Mapping, ICCV 2017

Structured light
Range 0.5-5 m, only indoor
Good support, but probably overcome by RealSense

Intel RealSense R200
○
○
○
○
○

●

Rotating ToF laser scanner
Range 0.2-50 m, both indoor and outdoor
Works reliably, but there is a lot of noise, shadows anddistortions close to the scanner
Needs to be actively tilted for full 3D scans

Asus Xtion Pro○○○

We’re preparing for solid-state lidars*

Sensors do not work in smoke, laser
scanner has problems with water, wetroads and linoleum

Sick LMS-300

Structured light + onboard IR stereo
Range 0.5-20 m, both indoor and outdoor
Indoor/outdoor switching not perfect
Noise with high variance in depth
Linux drivers not perfectly stable

ZED camera○○○

RBG stereo, computations out-of-board, requires CUDA
Range 0.3-20 m, both indoor and outdoor
Good results, but difficult to integrate (both HW and SW)

Real Deployment in Amatrice, Italy

Lessons Learned During TRADR, ROSCon 2017, 22 Sep. 2017
M. Pecka, S. Caccamo, R. Dubé , V. Kubelka, D. Reuter

Presentation available at bit.do/tradr

Real Deployment in Amatrice, Italy, Sep. 2016●●●

Strong earthquake
destroyed a big part ofAmatrice
TRADR UGVs and UAVs
were part of an assessment
mission in two damaged
churches
Robots provided detailed
3D models of the buildings
without risking human lives

That’s all folks!

Lessons Learned During TRADR, ROSCon 2017, 22 Sep. 2017
M. Pecka, S. Caccamo, R. Dubé , V. Kubelka, D. Reuter

Presentation available at bit.do/tradr

Released TRADR SW

Project Resources●

Web: tradr-project.eu
Youtube channel: goo.gl/F1Q4MG
Datasets (video+laser+TF, indexed bybagbunker): goo.gl/fC6sbo

Available on
github.com/tradr-project/tradr-doc○

Axis camera driver (Axis network camera
driver, written in Python)○

SegMatch (Loop-closure detection
algorithm)○

LaserSlam (End-to-end system tolaser-based graph SLAM user laser point
clouds)○Tensorflow ROS (An easy way how to linkTensorflow C++ API to ROS programs)○

Tracked vehicles in Gazebo (Added
support for tracked vehicles to Gazebo)

Lessons Learned During TRADR, ROSCon 2017, 22 Sep. 2017
M. Pecka, S. Caccamo, R. Dubé , V. Kubelka, D. Reuter

Presentation available at bit.do/tradr

Learning for Active 3D Mapping with Solid-State Lidar