ROS BUNDLING
MIKE PURVIS

THE HISTORY
ros-install-osx, the evolution of development and deployment at Clearpath

THE PROBLEM
stable releases for industrial customers in a rolling release world

THE BUNDLE
the tools which power our solution, and some unanticipated benefits

THE FUTURE
how the tooling can further evolve to better to support this deployment model

HISTORY

Building ROS Indigo on Mac OS X, two years ago.

The number of potential problems dwarfs the main body of the setup
instructions.
Actual installation instructions.

Stuff that might go wrong.

https://github.com/mikepurvis/ros-install-osx“This repo aims to maintain a usable, scripted, up-to-date installation procedure for ROS.
The intent is that the install script may be executed on a bare Yosemite or El Capitan
machine and produce a working desktop_full installation, including RQT, rviz, and Gazebo.”

Initial Gist
Mar 2014

June 2016
Brewed Python rosdeps

July 2015
Github Repo

July 2016
OS X Travis Tests

New approach used brewed python/numpy/scipy instead of the systemversions of those packages, and built in parallel with catkin_tools, rather thancatkin_make_isolated. Started life as a gist, eventually evolved into runnable
script. Positive side effect has been a centralized bug-tracker and place toreview and collaborate on fixes with a growing community of ROS Mac users.

rosinstall_generator desktop_full --rosdistro indigo--deps --tar > desktop_full.rosinstall
wstool init -j8 src desktop_full.rosinstallrosdep install --from-paths src --ignore-src --rosdistro indigo--as-root pip:no -y
catkin config —-install
catkin build
source install/setup.sh

Process is:
1. Generate a list of all packages which will be in the workspace.
2. Download the tarball for each package.
3. Install system dependencies using rosdep (stuff like CMake, Boost, PCL).
4. Giant parallel build!

This is what the build looks like in catkin_tools. We’re not going to dwell here
except to say that a catkin_tools is equivalent in approach to
catkin_make_isolated, meaning that each package is configured as its own
project, rather than being combined together. This is critical because plain
CMake packages like orocos_kdl and catkin itself can’t be combined into one
workspace the way catkin packages can.

- tar:

desktop_full.rosinstall

local-name: actionlib
uri: https://github.com/ros-gbp/actionlib-release/archive/release/indigo/actionlib/1.11.6-0.tar.gz
version: actionlib-release-release-indigo-actionlib-1.11.6-0- tar:
local-name: angles
uri: https://github.com/ros-gbp/geometry_angles_utils-release/archive/release/indigo/angles/1.9.10-0.tar.gz
version: geometry_angles_utils-release-release-indigo-angles-1.9.10-0- tar:
local-name: bond_core/bonduri: https://github.com/ros-gbp/bond_core-release/archive/release/indigo/bond/1.7.17-0.tar.gz
version: bond_core-release-release-indigo-bond-1.7.17-0...

This is a snippet of what’s inside the rosinstall file that wstool consumes. It’s alist of tarball links which will be downloaded and unpacked to populate thesource workspace.

actionlib:
release:ros/rosdistro/indigo/distribution.yaml

tags:
release: release/indigo/{package}/{version}
url: https://github.com/ros-gbp/actionlib-release.git
version: 1.11.6-0
bond_core:release:packages:- bond- bond_core

The rosinstall_generator tools uses as its source data the distribution.yaml
which governs which version of every package is the current latest in a given
distribution. So in this case, version 1.11.6-0 was the last bloomed release ofactionlib into ROS Indigo. The build farm uses this data to generate Jenkins
jobs to build debian packages, but rosinstall_generator uses it to generate
source workspaces.

- bondcpp- bondpy- smclib
tags:

actionlib:
release:ros/rosdistro/indigo/distribution.yaml

tags:
release: release/indigo/{package}/{version}
url: https://github.com/ros-gbp/actionlib-release.git
version: 1.11.6-0

The generator is capable of generating a list of git URLs, but downloading
tarballs is much faster than cloning git repos, so there is logic to recognize
known git hosts and supply tarball URLs with the --tar flag.

bond_core:release:packages:- bondhttps://github.com/ros-gbp/actionlib-release/archive/release/indigo/actionlib/1.11.6-0.tar.gz- bond_core- bondcpp- bondpy- smclibtags:

rosinstall_generator desktop_full --rosdistro indigo --deps --tar

This used to support github only, but we patched it to also recognize bitbucket
and gitlab. Now the other missing piece here is the --deps flag. How doesrosinstall_generator know which packages to pull in when you call for
dependencies?

http://repositories.ros.org/rosdistro_cache/<distro>-cache.yaml.gz

Every few minutes the buildfarm examines the distribution.yaml and updates asecondary yaml which is the contents of the distribution plus an additional
dictionary which contains the complete package.xml string for every package.
By accessing this cache, rosinstall_generator is able to quickly determine
recursive dependencies.

So. That’s ros-install-osx. The second piece of history is that Clearpath has
executed a pivot in the past two years, transitioning from a research products
company to focusing primarily on a full-stack solution for autonomous mobile
robots in the industrial logistics space. This has meant a bunch of changes in
how we deploy which are what have primarily driven the development of ourbundle pipeline.

THE PROBLEMS

About a year ago, I officially joined the industrial team, as Firmware andIntegration manager. Deployments to robots, developers, simulation
environments were all based on bloomed deb packages as generated bybuildbot-ros and then later ros_buildfarm. We had the following issues with thisapproach.

1
Rolling release. Robots and devs would all be running"testing debs" without any clear anchor on what versions

We had scripts which would use rospack list or rosversion to at least report theversions on a machine at a given time (for example, when a problem wasobserved, or a crash occurred), but knowing is only half the battle—it doesn’t
help you reproduce a bad state when trying to debug it later.

of packages they actually had.

Because ros_buildfarm sets up the test environment by installing the latest
released version of all test dependencies.
2
Coupled Repos. Having many interdependent packages
in different repos meant having to bloom changes all thetime to keep unit tests from breaking, rather thanbecause there was a logical, releasable change.

3
The release process was a debmirror snapshot of thetesting repo, similar to how upstream periodicallysyncs packages to public.

The easy argument here is “don’t have so many repos,” but this is not so simple,
particularly when you consider our relationship to upstream. Take our controls
code, which is spread across three repos: navigation, the planner, and thetracker, where navigation is a fork of upstream, and the planner and the tracker
are proprietary. At this point our navigation fork is permanent and permanently
private, but a year ago there was still a vague ambition to try to merge ourMeaning that there was no sane way to put incremental bugfixes on a build— you were either on the bleeding edge or a stale snapshot. We experimented
with injecting one-off bug fix debs into a release, but it was a very manual, errorprone process.

4
Meaningless package releases. For scale reasons, we strongly
preferred to run the test fleet on binaries, so to get code into
testing, developers would bloom broken stuff all the time.

Part of how I justify getting sucked into this project despite being the “firmware”
manager is that we increasingly view the entire robot as an embedded target to
be flashed, rather than a Linux PC on wheels with an ordinary computer
lifecycle.

THE BUNDLEThis was the proposal that my colleagues and I discussed as a starting point.

“Lets centrally build a workspace of a bunch of our stuff,
tar up the installspace, and use that on the robots.”Based on ros-install-osx— just build everything! Why bother separating out anoverlay?
stuff
all our“Lets centrally build a workspace of a bunch of our stuff,
tar up the installspace, and use that on the robots.”

We need versioning and a distribution scheme; let’s just use a debs and an apt
repo; those are already a thing.
stuff
all our“Lets centrally build a workspace of a bunch of our stuff,
tar up the installspace, and use that on the robots.”m a ke
a deb
packa
ge of

So your first question might be “seriously, a deb package, in this age of sexythings like containers?”
stuff
all our“Lets centrally build a workspace of a bunch of our stuff,
tar up the installspace, and use that on the robots.”
everywhere!
m a ke
a deb
packa
ge of

debootstrap trusty foo
chroot foo
docker build -t foo .
docker run -t -i foo /bin/bash
vagrant box add ubuntu/trusty64
vagrant up
vagrant ssh

virtualenv create foo
foo/bin/pip install -r requirements.txt

M O R E I S O L AT I O N

virtualenv create foo
foo/bin/pip install -r requirements.txt

C AT K I N

debootstrap trusty foo
chroot foo
docker build -t foo .
docker run -t -i foo /bin/bash

vagrant box add ubuntu/trusty64
vagrant up
vagrant ssh

Parallel XZ Compression
https://launchpad.net/~mikepurvis/+archive/ubuntu/dpkg

When using dpkg-deb to compress a 2.2GB payload, this version of dpkg-deb takes, on a 12-core machine:• 30 seconds to compress to 555MB (-z1)• 42 seconds to compress to 521MB (-z3)• 119 seconds to compress to 478MB (-z6)

Running the same test with dpkg 1.17.5 (trusty default):• 319 seconds to compress to 550MB (-z1)

And it’s true, isolation is a big thing right now. Type and degree of isolation
varies.

Base catkin is isolated in terms of filesystem only. Sort of a virtualenv for C++
CMake projects. We’re not doing multiple apps, and port or CPU isolation would
just get in our way, so we don’t need a docker container— we can ship this thing
as a deb, and then there’s a lot of additional tools we can take advantage of in
the debian ecosystem which don’t exist for docker (and we don’t have to run the
daemon).

If you’re like me and you suddenly find yourself generating a bunch of large-ish
debs, you’ll quickly discover that the default compression for dpkg-deb is singlethreaded xz, which can be extremely slow. Add this PPA to your 8 or 16 corebuildslave and watch it scream through that end-of-build compression step.

bundles.yaml

dsc

rosbundlerdeb

cowbuilder

deb
deb

bundles.yaml

dsc

rosbundlerdeb

cowbuilder

deb
deb

•Calls rosinstall_generator to get list of all packages in workspace.
•Generates debian/rules and debian/control (resolving rosdeps to system packages).
•Downloads all tarballs for workspace packages/repos•Calls dpkg-buildpackage -S

actionlib:
release:ros/rosdistro/indigo/distribution.yaml

tags:
release: release/indigo/{package}/{version}
url: https://github.com/ros-gbp/actionlib-release.git
version: 1.11.6-0
bond_core:release:packages:- bond- bond_core- bondcpp- bondpy- smclib
tags:

So, here’s basically our process. A bundle config file defines what our bundles
are, where the rosdistro for each one is, and what packages should be used to“seed” rosinstall_generator.

Our rosbundler tool generates the required debian package metadata files,
including instructions for how to build everything (the debian/rules file), andthen downloads the tarballs, naming them according to debian conventions.
Keen observers will note that this doesn’t actually address a lot of the problems
listed above. By default, rosinstall_generator is always grabbing the latest
released version of every package, so this bundle is equivalent to whatever thetesting debs are at the moment that the bundle is built. How do we actually lockin the versions of the packages?
Well, here’s the distribution file which actually defines the versions ofeverything. This lives in a git repo, and a git repo can be tagged.

actionlib:

ros/rosdistro/indigo/distribution.yaml

release:tags:

release: release/indigo/{package}/{version}

So we’ll tag the repo with the number of our major release, and that’ll forevergive us a reference point to regenerate this configuration.

url: https://github.com/ros-gbp/actionlib-release.git
version: 1.11.6-0

tag this repo

bond_core:release:packages:- bondto lock these versions- bond_core- bondcpp- bondpy- smclibtags:

In fact, we use a regular Debian changelog file which lives alongside thebundles.yaml file, and the version specified in each changelog entry refers
directly to a rosdistro tag.

rosbundles/debian/changelog

clearpath-2.0 (2.0.2-0) trusty; urgency=medium* [redacted]-- Mike Purvis <mpurvis@clearpathrobotics.com>

Thu, 23 Jun 2016 15:32:59 -0400

clearpath-2.0 (2.0.1-0) trusty; urgency=medium* [redacted]-- Mike Purvis <mpurvis@clearpathrobotics.com>

Wed, 22 Jun 2016 16:04:05 -0400

Big and bold so it can’t be missed.

Each major release of our entire software
stack is a tag on our rosdistro repo,
locking the version of every package.

bundles.yaml

dsc

rosbundlerdeb

cowbuilder

deb

So to finish this out, when we’re building a “release” bundle, we use the package
versions from a tag of our rosdistro, rather than the master branch.

deb•Checks out rosdistro repo at the branch/tag specified in the yaml config.
•Refreshes the rosdistro cache.
•Calls rosinstall_generator to get list of all packages in workspace.
•Generates debian/rules and debian/control (resolving rosdeps to system packages).
•Downloads all tarballs for workspace packages/repos•Calls dpkg-buildpackage -S

PRERELEASERELEASEversion is a timestamp

version from changelog

rosdistro from branch tip

rosdistro from tag

actionlib:
release:tags:
release: release/indigo/{package}/{version}
url: https://github.com/ros-gbp/actionlib-release.git
version: 1.11.6-0

And prior to tagging a legit “release”, we can still build a “prerelease”, which iseach package at its latest version. Eventually we may tag those as well— it
would be nice to understand which repos have mutated when we’re looking at aregression.

Now, we’ve talked a lot about the release stanza of the distribution.yaml entry—
this is the entry whose git URL references a GBP repo, one managed by bloom.

But the source stanza is actually where the bundle really comes into its fullpotential. Let’s look at a new rosinstall_generator flag.

actionlib:
release:tags:
release: release/indigo/{package}/{version}
url: https://github.com/ros-gbp/actionlib-release.git
version: 1.11.6-0
source:type: git
url: https://github.com/ros/actionlib.git
version: indigo-devel

Adding —upstream-development means that instead of getting the source fromthe bloom GBP repo, we get it directly from the source repo.
rosinstall_generator desktop_full --rosdistro indigo--deps --tar --upstream-development

rosinstall_generator- tar:
local-name: actionlib
uri: https://github.com/ros-gbp/actionlib-release/archive/release/indigo/actionlib/1.11.6-0.tar.gz
version: actionlib-release-release-indigo-actionlib-1.11.6-0- tar:

default

local-name: actionlib
uri: https://github.com/ros/actionlib/archive/indigo-devel.tar.gz
version: actionlib-indigo-devel--upstream-development

Above is what we get by default— this is the latest released source, pulled fromthe GBP repo. Underneath is with the —upstream-development flag, this is thedevelopment branch from the upstream repo.
Can you see what’s happening here? There’s a new type of bundle that’s
possible, where we don’t need to bloom at all.

The devel bundle is built entirely of devel sources. Seems scary right? This has
actually worked out phenomenally well for us, for a few reasons.
DEVEL

PRERELEASERELEASEversion is a timestamp

version is a timestamp

version from changelog

rosdistro from branch tip

rosdistro from branch tip

rosdistro from tag

upstream devel sources

DEVEL
version is a timestamp• test fleet is always running the latest devel code.
• when something breaks upstream (eg, ros_comm), wecatch it immediately.

rosdistro from branch tip• we’re motivated to push patches, since we get the fix
right away.

upstream devel sources• we bloom for logical functional releases releases only.

MAJOR VERSION• devel bundle can feed other testing workflows, including
multi-robot simulation testing and unit test jobs.

BUNDLE TYPE
devel

prerelease

release2.0

clearpath-2.0devel-robots
clearpath-2.0devel-nimbus
clearpath-2.0devel-sdk

clearpath-2.0pre-robots
clearpath-2.0pre-nimbus
clearpath-2.0pre-sdk

clearpath-2.0-robots
clearpath-2.0-nimbus
clearpath-2.0-sdk

2.1

clearpath-2.1devel-robots
clearpath-2.1devel-nimbus

clearpath-2.1pre-robots
clearpath-2.1pre-nimbus

clearpath-2.1-robots
clearpath-2.1-nimbus

clearpath-2.1devel-sdk

clearpath-2.1pre-sdk

clearpath-2.1-sdk

2.2

clearpath-2.2devel-robots
clearpath-2.2devel-nimbus
clearpath-2.2devel-sdk

clearpath-2.2pre-robots
clearpath-2.2pre-nimbus
clearpath-2.2pre-sdk

clearpath-2.2-robots
clearpath-2.2-nimbus
clearpath-2.2-sdk

A P T LY

In terms of our actual infrastructure, we build the bundles on Jenkins, and thensend them to Aptly. Aptly is super cool for many reasons— two of them are thatit can run as a REST service, so you don’t have to deal with SSH or SCP, you canjust POST packages to it with curl if you want.

JENKINSThis is our build status page, which is source repo oriented as opposed to thebuild farm one, which is deb package oriented.

Each row pops open to reveal the changelog and commits which have occurred
between the last release and the current level branch.

$ rosdistro_freeze_source -hFreeze a rosdistro's source branch versions to hashes or tags. Ifneither--release-version nor --release-tag are specified, the hashes of thecurrent
devel branches are used.

positional arguments:
index

Path to a local index.yaml file.

dist_names

The names of the distributions (default: all)optional arguments:-h, --help

show this help message and exit

-j JOBS, --jobs JOBS

How many worker threads to use.

-q, --quiet
usage).

Suppress updating status bar (for script/CI--release-version

Freeze to the hash of current release tag.

--release-tag

Freeze to name of current release tag.

rosdistro_freeze_source

actionlib:
source:type: git
url: https://github.com/ros/actionlib.git
version: indigo-devel

rosdistro_freeze_source

actionlib:
source:type: git

actionlib:
source:url: https://github.com/ros/actionlib.git
version: 32ce9fdb3194228795a2f31faee8259dd57ae452

type: git
url: https://github.com/ros/actionlib.git

actionlib:
source:version: indigo-devel
type: git
url: https://github.com/ros/actionlib.git
version: 1.11.6

Once a major release for us goes into maintenance, we “freeze” the devel
branches to point to the tag of the last release. This allows development tomove forward, while still giving us a stable level build upon which to put hot
fixes.

D E M O N S T R AT I O N
https://github.com/mikepurvis/ros-bundling

catkin test, catkin docs.
dropping release stanzas and blooming, consi.
once the ubuntu store is open source, investigate packaging as a snap.
THE FUTURE

catkin docs

• implement as catkin_tools plugin• make rosdoc_lite into plain python package, so theplugin could depend on it.
• allow building all docs in one big workspace, using thesame parallel executor model as catkin build.

One of the big wins with this approach would be potentially giving more
opportunities to supply global overrides. So, same as how building the bundle
lets you do fun things like set compiler flags that correspond to the architectureof your robot PC, or archive debug symbols, a future catkin docs might allow
injecting a Doxygen theme, etc.

https://github.com/catkin/catkin_tools/issues/381

catkin test

• implement as catkin_tools plugin• safe parallel testing of a workspace which may containcmake and catkin (and ament?) packages.
• have a scheme for tests to declare and then properly
mutex shared testing resources, like the Gazebo port.
• would work for CI, but also be a better story locally fordevs using catkin_tools.
https://github.com/catkin/catkin_tools/issues/397

source only

• drop release stanzas from internal distribution yaml.
• only bloom packages which are going for public release,otherwise do only catkin_prepare_release to uprev
package.xml and create tag.
• depends on source manifest caching for dependency
resolution.

One of the other major functional gaps in catkin_tools right now is not having agood story for running tests.

• drop release stanzas from internal distribution yaml.

source only

• only bloom packages which are going for public release,otherwise do only catkin_prepare_release to uprev
actionlib:

package.xml and create tag.

source:type: git
url: https://github.com/ros/actionlib.git
version: indigo-devel
release_version: 1.11.6

Snaps• Packaging as a squashfs is compelling.
• Delta updates are compelling.
• Multiple release streams (edge, RC, etc) is very
compelling.
• Blocked on the repository code being open sourced.

THANKS
William and Dirk
Jon Bohren
Nikolaus Demmel
Mike Ferguson

Contributors to ros-install-osx
Clearpath

https://ottomotors.com/company/careers

OSRF staff for maintaining the tools and entertaining my crazy ideas and PRs.
Jon for re-architecting the catkin_tools backend.
Nikolaus for helping with OS X issues, and contributing the no-sudo option torosdep.
Mike Ferguson for buildbot-ros and getting me into debian packaging andcowbuilder.
Clearpath for being an awesome place to work, and for allowing me time towork on this, and especially my colleagues on the platform and tools/
infrastructure teams. Obviously, we are hiring.