Building a Computer Vision Research Vehicle with ROS
ROSCon 2017 | 2017-09-21 | Vancouver
Andreas Fregin, Markus Roth, Markus Braun, Sebastian Krebs & Fabian Flohr

Agenda1. Introduction
2. History
3. Triggering a Heterogeneous Sensor Setup
4. Our Calibration Solution
5. Enhancing ROS Tools / Handling Data
6. Q&A

Daimler AG

Building a Computer Vision Research Vehicle with ROS | Andreas Fregin | 2017-09-21 | Page 2

About Us• Daimler is the corporate parent of Mercedes-Benz.

• The authors started in team „Pattern Recognition and Cameras“ as PhDs.
• Main research topics: pedestrian intention recognition, traffic light recognition.
• Interests: object recognition from camera images, machine learning.

• Using ROS as research framework for computer vision.

Sebastian Krebs

Daimler AG

Markus Braun

Andreas Fregin

Markus RothFabian Flohr

Building a Computer Vision Research Vehicle with ROS | Andreas Fregin | 2017-09-21 | Page 3

2011/12

How I came to ROS• University• RoboCup@Work

2015

Daimler• Need for a framework• Used est. automotive framework• Missed simplicity, introspection andespecially the doc. (wiki) of ROS• Came back to ROS

RoboCup@Work: Basic Transportation Test
Daimler AG

Precision Placement Test / League Winners

Building a Computer Vision Research Vehicle with ROS | Andreas Fregin | 2017-09-21 | Seite 4

Daimlers History in ADAS & Autonomous Driving Research

ActiveSafetyBlind Spot WarnerStop & Go Assist
Lane Departure Warning
Emergency Call
Speed Limit Assist
Speech In/Output
Emergency Breaking Assist
Head-up
Display
Attention Assist
Adaptive Cruise Control
Night View Lane Keeping Assist

Digital Map

RDS-TMC

Traffic
Dynamic Navigation
Management
Travel Information Services

AutonomousDriving

Daimler AG

Adaptive LSA

Mobile Service “PTA”
Strategy Management
Online ServicesFloating Car Data…
Building a Computer Vision Research Vehicle with ROS | Andreas Fregin | 2017-09-21 | Page 5

Our ROSified Research Vehicles

Daimler AG

Building a Computer Vision Research Vehicle with ROS | Andreas Fregin | 2017-09-21 | Page 6

Universal CAN Message Decoder• Message generator for CAN-bus messages
auto. generation

gencpp, genpy, genlisp, …

.msg files

CAN msg.
description

client library msgs

• Decoding/Publishingw221_body_can/velocity
w221_body_can/yaw_rate

raw CAN msg.
CAN hardware

decoder

w221_body_can/stw_angle
w221_body_can/radar_objects…

Daimler AG

Building a Computer Vision Research Vehicle with ROS | Andreas Fregin | 2017-09-21 | Page 7

Enabling Low-Level Sensor Fusion
Target• Capture surrounding at the same moment in time
• … across different sensors• Precisely time-stamp sensor-readings
Constraints• Heterogeneous sensors• Different sensor nodes• Maybe different cycle rates• Unstamped sensor data from CAN-bus

Daimler AG

Building a Computer Vision Research Vehicle with ROS | Andreas Fregin | 2017-09-21 | Page 8

Software Triggering• Use the host PC to software-trigger all sensors

CAN

bag_rec
Sensor Data

camera1
camera2

CAN Busses

camera3
camera4

Daimler AG

Building a Computer Vision Research Vehicle with ROS | Andreas Fregin | 2017-09-21 | Page 9

Hardware Triggering• Use a trigger generator to hardware-trigger all sensors• Same acquisition time – but when?

CAN
Sensor data
Trigger signal

Unstamped data
Stamped data
bag_rec

camera1
camera2
camera3
CAN Busses
Daimler AG

camera4
Building a Computer Vision Research Vehicle with ROS | Andreas Fregin | 2017-09-21 | Page 10

Sensors do not know about reference time
• The exposure was triggered at the exact same moment, so the data (images) show thesame content• Processing time of heterogeneous setups will vary ->data (images) arrive at different

moments in time
• Timestamping using ros::Time::now() will result in different timestamps• Timestamping using ros::Time::now() is not correct (arrival vs. acquisition!)• We need to know the moment of triggering in reference time

Daimler AG

Building a Computer Vision Research Vehicle with ROS | Andreas Fregin | 2017-09-21 | Page 11

PTP Time-Sync (Precision Time Protocol, IEEE1588)• Trigger (Microcontroller) does not know about reference time

• Time-Synchronization: STM32F4 + LWIP + ROSUDP + PTPd
• With each trigger signal, also a trigger message is generated• Publishing trigger message (std_msgs/Header)• Sensor nodes receive the trigger message before the sensor data arrives: proper

timestamped images, while ensuring all different sensor data have the exact sametimestamp!
• Result: All camera images show the exact same moment AND we know the timestampof that moment
Daimler AG

Building a Computer Vision Research Vehicle with ROS | Andreas Fregin | 2017-09-21 | Page 12

Hardware Triggering with known time
• Microcontroller does know about reference time (via PTP from PC)• Microcontroller publishes trigger as ROS std_msgs/Header

Unstamped data
Stamped data

Sensor data
Trigger signal
Trigger ROS topic

CAN

STrigger

bag_rec

camera1
camera2
camera3
CAN Busses
M
Daimler AG

camera4
Building a Computer Vision Research Vehicle with ROS | Andreas Fregin | 2017-09-21 | Page 13

Synchronization with Velodyne LiDARs• Microcontroller is now PTP time master using GPS time (NMEA string parsing)• PC is PTP time slave
M

Unstamped data
Stamped data
CANvelodyne

Sensor data
Trigger signal
Trigger ROS topic

GPSTrigger

bag_rec

camera1
camera2
camera3
CAN Busses
SDaimler AG

camera4
Building a Computer Vision Research Vehicle with ROS | Andreas Fregin | 2017-09-21 | Page 14

Calibration• Whenever you fuse data you need to know about times AND coordinate frames• Even small errors (sub-decimal) in orientation result in huge position errors for distantobjects• We need a good extrinsic calibration
Cameras• Laser scanners• …
•

Daimler AG

Building a Computer Vision Research Vehicle with ROS | Andreas Fregin | 2017-09-21 | Page 15

Intrinsic Camera Calibration• cameracalibrator.py comes with OpenCV checkerboard-detector• Has a informative UI that teaches you where to hold the checkerboard (X/Y/Size)• Does pick the images from running video: user doesn‘t has the chance to hold still toavoid motion blur, etc.
• Does not allow to modify data that is used for the calibration step
• Does not generate a sensor-to-car transformation

Daimler AG

Building a Computer Vision Research Vehicle with ROS | Andreas Fregin | 2017-09-21 | Page 16

Calibration Requirements• One-man show

• On demand checkerboard detection• Live detection inspection• Remove images• Add specific images

Daimler AG

Building a Computer Vision Research Vehicle with ROS | Andreas Fregin | 2017-09-21 | Page 17

Server-Client Calibration using Car-PC + Linux-Tablet
Topicstf

total_st.
cameraimage{_left/_right}

cal-servercal-server

Daimler AG

cal-gui

Trigger detection
Trigger calibration

ServicesCarimage(_sbs)/compresseddelete detection
load data
save datacal-gui

TabletBuilding a Computer Vision Research Vehicle with ROS | Andreas Fregin | 2017-09-21 | Page 18

Timeshift Recording• Good example for ROS-tool enhancement• Start recording in the past
• RAM Buffer• Trigger topic (delayed start/stop)

Seconds of rec.

Minutes of rec.

Hours of rec.

Rosbag player enhancements• Step-topic (play/pause)• Triggered playback
Daimler AG

Building a Computer Vision Research Vehicle with ROS | Andreas Fregin | 2017-09-21 | Page 19

Rewriting Rosbags• Don’t be afraid using rospy.rosbag to modify existing ROSbags• add sensor data
• add TF (Calibration)• add ground truth• correct data (e.g. frame_ids, image_encodings, …)• …

Daimler AG

Building a Computer Vision Research Vehicle with ROS | Andreas Fregin | 2017-09-21 | Page 20

Powerful tools / packages• Setting up complex image processing setups using nodelets• Strongly typed messages lead to node exchangeability•

Example: different detectors all use the same in/output messages• Extremely powerful packages like image_geometry speed up research• Having tf as the transformation central• Launch system is very helpful (especially including other launch files)• Diagnostics capabilities, …
Daimler AG

Building a Computer Vision Research Vehicle with ROS | Andreas Fregin | 2017-09-21 | Page 21

Our lessons learned• ROS already includes the concepts to realize complex, heterogeneous sensor setups• ROS can handle high data throughput and high cycle rates• ROS is a good starting point for handling large data
• If your needs exceed what ROS comes with – extend it!

Daimler AG

Building a Computer Vision Research Vehicle with ROS | Andreas Fregin | 2017-09-21 | Page 22

Questions ?

Daimler AG

Building a Computer Vision Research Vehicle with ROS | Andreas Fregin | 2017-09-21 | Page 23