The ROS build farm
What it can do for me(And how it does it)
Oct. 9th 2016
Dirk Thomas
ROSCon 2016, Seoul, Korea

build.ros.org

Create easy-toinstall packages

Run tests before /after code changes

Generate API doc
and meta information

Central Service vs. Distributed
You could
create Debian packages yourself
run the test using e.g. Travis CI
generate API documentation yourself
But
setting up all these processes is quite some effort
having someone else do it is great for the users
the artifacts should be hosted in a central placeif you download "official" binaries you know what you are getting

Central Service But Distributed Repositories
A ROS distribution consists of numerous packages
various vcs types
various hosting services

github.com/ros/rosdistro
for each ROS distribution
target platforms

rosdistro/index.yaml
distributions:...
kinetic:
distribution: [kinetic/distribution.yaml]
distribution_cache: <url to kinetic-cache.yaml.gz>

e.g. Kinetic: Ubuntu Xenial and Wily,
Debian Jessie, (Fedora 23 and 24)

packages and versions
referencing all repositories
format of yaml files specified in REP 143

To use any of these services
you need to register your repositories

rosdistro/kinetic/distribution.yaml
repositories:...
catkin:doc:...
release:...
source:...
release_platforms:...
ubuntu:- wily- xenial

The High-Level View
Get the source code:e.g. git clone
Install the required dependencies:
using rosdep
Build a package:
cmake, make, make install
Run the tests:
make test
Create a Debian package:
apt-src build
Generate documentation
rosdoc_lite

Some More Details
Need to setup the environment for building
Install all dependencies (using binary packages)
Since each job has different dependencies
each job needs to start with a "clean" state
Inside a Docker container
Build the code from one repository
Might contain more than one ROS package
One container for
The repository is cloned into a catkin workspace multiple packages
Invoke catkin_make_isolated
Install the codeTo check that the install step "works"
Build vs. test
Build and run the tests
dependencies
Each test produces a xUnit-like result file

catkin_make vs. catkin_make_isolated
While the Docker container contains the dependencies for all packages
Each package is still built "in isolation":catkin_makecatkin_make_isolated
CMake calls

1, for the whole ws

N, each pkg separately

Side effects

 Need to depend on other pkgs targets
catkin_EXPORTED_TARGETS−

Location ofartifacts

Merged in a single folder
E.g. one include dir

Each pkg has its ownfolder

Side effects

 Package can implicitly accesse.g. the headers from other pkgs−

Faster due to higherparallelization

Clean separation, betterto identify problemsPros The build farm only builds single threadedRun tests before /after code changes

Devel Jobs
 check 1h

IN: repository
rosdistro/kinetic/distribution.yaml

pkg/package.xml+ commiters since the job was stablebuild
installinstall test dep

OUT: emailbuild tests

OUT: result

run tests

CMake / compiler warnings, failing tests

Docker #2<maintainer
email="dthomas@osrfoundation.org">Dirk Thomas</maintainer>

install build dep

Docker #1

genmsg:source:type: git
url: https://github.com/ros/genmsg.git
version: <valid branch name>clone

Pull Request Jobs
GitHub webhook
IN: repository
rosdistro/kinetic/distribution.yaml
genmsg:source:test_pull_request: true
type: git
url: https://github.com/ros/genmsg.git
version: <valid branch name>

OUT: GitHub PR status

Pull Request Jobs
Limitations
build.ros.org only supports GitHub at the moment
Jenkins GitHub user ros-pull-request-builder needs
access to the org unit
wiki.ros.org/buildfarm/Pull request testing
Alternatives
Run the same process using any other CI provider
E.g. Travis CI, see example .travis.yml file:docs in github.com/ros-infrastructure/ros_buildfarmGenerate API doc
and meta information

Doc Jobs check 3d

IN: repository
rosdistro/kinetic/distribution.yaml
genmsg:doc:
type: git
url: https://github.com/ros/genmsg.git
version: <valid branch or tag name>clonechangelog
rst → html

meta information
cross ref index

OUT: generated docs
upload to webserver

install doc dep

OUT: email

generate msg / srv

OUT: result

rosdoc_lite ∀ pkgs

rosdoc_lite warnings

Docker

OUT: update cross ref indexWiki Integration

wiki markup<<PackageHeader(roscpp)>>Create easy-toinstall packages

Release Processsource repository

catkin_prepare_release tag version

release repositorybloom each package, with Debian files

Debian source packages

source jobs
 each os name and os code namee.g Ubuntu Xenial

Debian binary packages

binary jobs
 each os (code) name and arche.g Ubuntu Xenial amd64

Bloomon a single branch:- pkgA- pkgB

package.xml:- build type- build deps
- run deps
CHANGELOG.rst

AUTOMATION



release repository
separate branches:

pkgA

pkgB

pkgAe.g. Debian- changelog- control- rules

rst → custom- Makes releasing as easy as possible- But allows customization anywhere in the process- Creates the rosdistro pull request for you

Decide how to build pkg

pkgA

Map rosdep keys totarget specific names

source repository

Each target platform needs custom specialization, e.g. Wily vs. Xenial

Sourcedeb Jobs
 check 15min

IN: release repoclone
specific branchpkg, os (code) namee.g. debian/ros-kinetic-bar-1.2.3-4_xenial

OUT: Debian source package
upload to apt repo

OUT: email
OUT: result

install tools
debhelper
git-buildpackage

gbp buildpackage

Docker

OUT: trigger binary jobs
∀ architectures

Binarydeb Jobs
triggered by sourcedeb job
IN: Debian source package

fetchfrom apt repo

abort ?
append timestamp
OUT: trigger binary jobs

OUT: Debian binary package
upload to apt repo

OUT: email
OUT: result

install build dep
dpkg-buildpackage

Docker∀ downstream pkgs

Apt Repositories"main"

packages.ros.org/ros/ubuntu"testing""testing"

manual sync"main"

packages.ros.org/ros-shadow-fixed/ubuntu"building"auto sync

Distributed mirrors
newly built pkgs go here

Server colocated with Jenkins
Please do not use
this one directlystild

ing

Je

ild
in
nk g sin qus b arin e liar nk
yjo tobbu

im ng
ai
ntebu

Status Pages

repositories.ros.org/status_page/

"Decipher" Jenkins Job Names
Some example job names
Isrc_uS__roscpp__ubuntu_saucy__source
Jdev__ros_comm__ubuntu_trusty_amd64
Kbin_uX64__rospack__ubuntu_xenial_amd64__binary
Until first __: prefix for group of jobs

Until next __: the package or repository name
Then os name, os code name, arch (except for source jobs),
source / binary (for release jobs)

Jenkins Statistics
All
Ibin_arm_uThf
Ibin_uS32
Ibin_uS64
Ibin_uT32
Ibin_uT64
Idev
Idoc
Ipr
Isrc_uS
Isrc_uT
Kbin_dj_dJ64
Kbin_djv8_dJv8
Kbin_uW32
Kbin_uW64
Kbin_uX32
Kbin_uX64
Kbin_uxhf_uXhf
Kdev
Kdoc
Kpr
Ksrc_dJ
Ksrc_uW
Ksrc_uXManagecount successunstablefailureabortednot_built
disabled
39785 38504 (96.78 %) 148 (0.37 %) 361 (0.90 %) 250 (0.62 %) 14 (0.03 %)
508 (1.27 %)2348
1997 (85.05 %)16 (0.68 %)
8 (0.34 %)0 (0.00 %) 327 (13.92 %)
2348
2316 (98.63 %)
22 (0.93 %)
9 (0.38 %)
1 (0.04 %)0 (0.00 %)2348
2314 (98.55 %)
22 (0.93 %)
11 (0.46 %)1 (0.04 %)0 (0.00 %)2348
2333 (99.36 %)
12 (0.51 %)
3 (0.12 %)0 (0.00 %)0 (0.00 %)2348
2334 (99.40 %)
11 (0.46 %)3 (0.12 %)0 (0.00 %)0 (0.00 %)769
645 (83.87 %)
26 (3.38 %) 98 (12.74 %)0 (0.00 %)0 (0.00 %)0 (0.00 %)
807
756 (93.68 %)
31 (3.84 %)
16 (1.98 %)1 (0.12 %)3 (0.37 %)0 (0.00 %)29
20 (68.96 %)
3 (10.34 %)2 (6.89 %)0 (0.00 %) 4 (13.79 %)0 (0.00 %)2348 2348 (100.00 %)0 (0.00 %)0 (0.00 %)0 (0.00 %)0 (0.00 %)2348 2348 (100.00 %)0 (0.00 %)0 (0.00 %)0 (0.00 %)0 (0.00 %)
941
894 (95.00 %)
12 (1.27 %)
33 (3.50 %)0 (0.00 %)2 (0.21 %)
941
875 (92.98 %)
14 (1.48 %)33 (3.50 %)0 (0.00 %)19 (2.01 %)
941
903 (95.96 %)
12 (1.27 %)
26 (2.76 %)0 (0.00 %)0 (0.00 %)
941
903 (95.96 %)
12 (1.27 %)
26 (2.76 %)0 (0.00 %)0 (0.00 %)
941
910 (96.70 %)
10 (1.06 %)21 (2.23 %)0 (0.00 %)0 (0.00 %)
941
910 (96.70 %)
10 (1.06 %)21 (2.23 %)0 (0.00 %)0 (0.00 %)
941
817 (86.82 %)
11 (1.16 %)
30 (3.18 %)0 (0.00 %)83 (8.82 %)291
215 (73.88 %) 47 (16.15 %)
29 (9.96 %)0 (0.00 %)0 (0.00 %)0 (0.00 %)304
288 (94.73 %)
11 (3.61 %)
4 (1.31 %)
1 (0.32 %)0 (0.00 %)0 (0.00 %)30
23 (76.66 %)
3 (10.00 %)1 (3.33 %)0 (0.00 %) 3 (10.00 %)0 (0.00 %)
941
933 (99.14 %)
0 (0.00 %)0 (0.00 %)0 (0.00 %)8 (0.85 %)
941
941 (100.00 %)0 (0.00 %)0 (0.00 %)0 (0.00 %)0 (0.00 %)
941
941 (100.00 %)0 (0.00 %)0 (0.00 %)0 (0.00 %)0 (0.00 %)8686 (100.00 %)0 (0.00 %)0 (0.00 %)0 (0.00 %)0 (0.00 %)0 (0.00 %)Run "Jobs" Locally
The build farm is powered by the Python package ros_buildfarm
After installing it:
Debian package python-ros-buildfarm
PyPI ros_buildfarm
You can run:generate_devel_script.py <many args>
generate_doc_script.py <many args>
generate_release_script.py <many args>
to easily reproduce results of the build farmrun "jobs" for any target platform (within Docker) locallyTest Across Multiple Repos① You make related changes in different repos, e.g.
repo A: add a new API
repo B: use the new API→ The devel / PR jobs won't work for this case ☹
repo A: passes but doesn't really check the new API
repo B: fails because it's using the last released version of A
② You want to check if your changes break any downstream packages
repo A: make some changes
repo B: check that the tests for the last released version still pass
→ The devel / PR jobs don't cover for this case either ☹

Prerelease "Job"overlay workspace
Selected source repos
Deps from Debian pkgs
Selected overlay repos
underlay workspace

Implicit overlay repos

prerelease.ros.org

Supported Targets
The build farm can produce binary packages for
Ubuntu, Debian
amd64, i386, arm64, armhf
We can't cover all combinations (), for Kinetic we coverUbuntu: Xenial & Wily amd64 & i386, Xenial arm64 & amrhf
Debian: Jessie amd64 & arm64
Bloom also generates meta information for rpm (Fedora)
But we don't build binary packages (yet?)If you want to create binaries for a not supported target
Talk to us (<hint> donations can cover the cost for the needed resources )Contribute code to automate building binaries for new targets
or [see next slide]Your Own Build Farm
Use cases
Build a different OS name / OS code name / architecture combination
Process private repositories
You will need your own rosdistro database
Try modified version of the build farm
Two step process [wiki.ros.org/buildfarm]
1. Provision machines:
Jenkins master, Apache webserver, Build slaves
2. Generate the Jenkins jobs
Many configuration option

Expect continuous effort

What Can You Do
Register your repositories
add doc entry to generate documentation
create a wiki page for your packages
add source entry for devel jobs and easy cloning for users
if the devel job fails, don't remove the entry, only disable the job:
test_commits: false

e.g. your repo depends on other packages which are not released
consider enabling pull request testing
release your packages, also into newer ROS distros
Watch out for notification emails from Jenkins
read it, search for the problem, try to fix it, ask for help
as a last resort: disable the job, blacklist target, remove the releaseTroubleshooting"It works locally but the job fails" → something must be different
Have you tried using `catkin_make_isolated`?
Most commonly you have a dependency installed but the build farm doesn't.
Check the declared dependencies.
CMake Error at /opt/ros/kinetic/share/catkin/cmake/catkinConfig.cmake:83 (find_package):
Could not find a package configuration file provided by "<pkg_name>""The devel job passed but the release failed"In a devel job all packages share the same container. One package might
declare a dependency another package uses without declaring it itself.

Expect (very rarely) all kinds of "hicups"
apt update failing since the repository is being modified concurrently
network problems, GitHub resetting the connection, an http request not being
handled, ...
If your problem persists it's probably not such a hicup

Questions...

For more information go to:github.com/ros-infrastructure/ros_buildfarm