Lessons learned building a self-driving car onROS

Nicolò Valigi (nicolovaligi.com)
Software Engineer at Cruise Automation
September 29, 2018

1

The carA fleet of 100 self driving cars driving around San Francisco and ateam of 500 Software Engineers writing code for them.
2The sensors

Each car has ~ 10 cameras (fish-eye, wides, …), an handful of
LIDARs and radars, an IMU, wheel sensors, … Roughly 1GB/s ofdata coming in from the sensors.
3

Goals of this talkThe goal of this talk is to share some of the directions where wehave pushed ROS to make it scale up to a complex problem.
Three areas of focus:• Determinism and simulation• Core framework improvements• Developer tooling4Simulation and determinism

Why simulation

Testing on real cars is inconvenient and expensive.
We record sensor data as the fleet is driving, and we want to reuse
it for tests, simulations, and development of algorithms.
The idea is to spin up a bunch of cloud machines for large-scale
simulations of different scenarios.
but…

5Where ROS falls short

ROS supports playing back recorded data from a bag, essentiallyreplacing the driver nodes.
The C++ and Python client libraries can also listen to the recorded
clock ticks, essentially reproducing the original time stamps.
Simulating time is enough for small systems, but breaks down soon:• on slower hardware, you either slow down the replay oroccasionaly miss frames.
• determinism goes out of the window (see Ingo Lütkebohle’s
great talk at ROSCon 2017).

6

What we did at Cruise

Simulation needs to be a first-class concept of any Robotics
framework. On top of ROS, we added systems to achieve
deterministic lock-step simulation from recorded data.
Three steps to achieve this:
1. remove flexibility in the pub-sub model → precise
computation graph2. schedule computations deterministically (by controlling dataflow)
3. send acknowledgements to keep track of computations inside
the nodes.
7Core framework improvements

Interaction with the OS scheduling
ROS encourages a proliferation of nodes and nodelets. Under load,
this becomes an OS scheduling nightmare, as you have many more
processes/threads than CPU cores.

8

Runtime profiling

The framework doesn’t help you keep track of the runtime ofdifferent components. In the typical Robotics stack, this meansthat downstream components are late and you don’t know why.
What we did at Cruise• built some tooling around the Chrome trace viewer forvisualizing flame graphs throughout the system.
• pass around contextual information that can be used to trace
metrics about a thread of computation (e.g. computer visionpipeline).

9Profile viewer

10Developer toolingRosbag, serialization, and dataBags are good for logging but:• the file format needs lots of file seeks → unsuitable for“big-data” style tooling• ROS serialization is not forwards compatible at all: justadding a new field breaks deserialization on the old code.
What we did at Cruise• built an extensive data pipeline to convert bags to slice anddice bags, supporting big-data tools for batch requests.

11

Webviz vs ROS-style tooling

RQt, Rviz, and friends are very flexibile, but:• need a Linux workstation with ROS• have a steep learning curve for both use and customization
What we did at Cruise• created a Web-based frontend that emulates most of the RQt
set of features, and can easily be extended in Javascript.

12

Webviz13Build system

Catkin’s modularity means that it needs to do extra work for eachbuild, making it slow.
What we did at Cruise• migrated to Bazel, the open source version of Google’s buildsystem• sped up workstation builds and added a shared cache for CI

14

Thank you