The ROS Build Farm and You!
How the packages that you release becomebinary packages in the ROS repositories

whoamiSteven! Ragnarök
ROS Infrastructure Team
discourse.ros.org/u/nuclearsandwich
github.com/nuclearsandwich
twitter.com/nuclearsandwich
steven@openrobotics.org

Creating ROS 2 Packages
1

ros2 pkg create …

2catkin_generate_changelog

3

catkin_prepare_release4

Add a source repository entry in ros/rosdistro

5

Create a ros2-gbp release repository

6

bloom-release …

7

WAAAAAAAAAAAAAAAAAAIT

8

apt-get install ros-rolling-yourpackage

Steps 1, 4, and 5 are only required for initial setup.

What is "the build farm"?
A cluster of machines which communicate with acentral Jenkins build server to perform various build andautomation tasks for the ROS project and community.
There are actually two∗ of them.
build.ros.org
ROS Noetic
ROS Melodic
ROS Lunar
ROS Kinetic

build.ros2.org
ROS 2 Rolling
ROS 2 Humble
ROS 2 Galactic
ROS 2 Foxy

∗ci.ros2.org is specialized and not a usual build farmWhat is "the build farm"?
There are several different machine roles in the cluster:
Exactly one Jenkins host.
Exactly one Repository (repo) host.
At least one Agent host per architecture.
Optionally one or more CI Agent hosts.

What does the ROS build farm do?
Runs per repository CI
Tests pull requests and updates to the source branch.
Builds documentation
API and long form documentation for configuredpackages.
Builds packages for ROS repositories
Source packages and binary, packages.
Performs distribution management automation
Synchronize packages between repositories and update
repository status pages.

…and it looks good doing it!
The entire build farm process is…
• publicly accessible to the entire ROS community• run using open source software• using public configuration information ∗• running on hosts provisioned using open source
infrastrucure as code
∗excepting secrets like GPG private keys

ROS build farm open source projects
github.com /ros-infrastructure / ros_buildfarm
ros_buildfarm_config
reprepro-updater
cookbook-ros-buildfarm
ros2 /
ros_buildfarm_config
ros /rosdistroWhat are the ROS Repositories?
Package repositories hosted on packages.ros.org andthe ROS build farms.
APT repositories for Debian and Ubuntu
RPM repositories for Enterprise Linux, and perhaps
someday Fedora and/or OpenSUSE
ROS and ROS 2 have separate repositories although
there is no RPM repository for ROS 1.

What are the ROS Repositories?
Package repositories hosted on packages.ros.org andthe ROS build farms.
PublicInternal
ROS (1) ros ros-testing ros-building
ROS 2 ros2 ros2-testing ros2-buildingbuilding repositories are not publicly mirrored on packages.ros.org.

Builld farm repository "flow"
ros_bootstrap repository
Entry point for infrastructure packages and third party
packagesbuilding repository
Staging ground for packages being rebuilt
testing repository
Repositories for use in CI and manual testing
main repository
Default installation path for supported platforms

Don’t Panic
We’re going to get through this together

One step at a time…
Prepare the source release
2 Let the build farm do its thing
3 Wait for a sync
4 Install your package
11. Preparing the source release1. Preparing the source release
There are many infrastructure tools to help with this:1 catkin_generate_changelog to help create andupdate changelogs.
2 catkin_prepare_release to update version info, tag,and push the release
3 bloom-release to update the release repository data
for new releases.

1. Preparing the source releaseros/rosdistro release PRs opened automatically by Bloom.

2. The build farm's thing2. The build farm's thing
Jenkins release management jobs
1 rosdistro_cache jobs run every five minutes pollingfor changes in ros/rosdistro
2 reconfigure jobs for changed distributions aretriggered by rosdistro cache
3 every 15 minutes trigger jobs start packaging jobs
for changed packagesrosdistro cachegzip compressed yaml document containing
distribution_file

a copy of the distribution file from ros/rosdistro

package.xml contents from thecurrently released package version

release_repo_package_xmls

source_repo_package_xmls

empty (fetching source package.xml istricky)
namethe short name of this rosdistro

type, version

REP-141 format information

rosdistro-cache jobs
Examples rolling_rosdistro-cache, humble_rosdistro-cache• Generates a new rosdistro cache file and uploads it to therepository host.
• Runs on five minute intervals.
• Polls for distribution.yaml changes via HTTPS. Changes
sometimes lag due to CDN caching.
• Triggers reconfigure jobs for updated packages.

release reconfigure jobs
Examples Rrel_reconfigure-jobs, Rrel_rhel_reconfigure-jobs,
Rrel_ujv8_reconfigure-jobs• One job per release build file in ros_buildfarm_config• Uses release build config and ros_buildfarm templates togenerate a sourcepkg and binarypkg job for each package• Runs when triggered by rosdistro cache to update changedpackages• Runs every 24 hours to keep job configurations up to daterelease trigger jobs
Examples Rrel_trigger-jobs, Rrel_rhel_trigger-jobs, Rrel_ujv8_trigger-jobs• One job per release build file in ros_buildfarm_config• Compares packages in the building repository with expected packages
based on rosdistro cache and triggers sourcepkg and/or binarypkg jobs toimplement changes
Job parameters:
missing only – triggers jobs for packages not present in repositories.
source only – triggers jobs for source packages only; useful when source
packages need to be updated before binaries are rebuilt

Packaging jobs

source package jobs
Examples Rsrc_uJ64__rclcpp__ubuntu_jammy_source• Fetch package sources with platform-specific metadata from ros2-gbp
repository and create platform source package (.dsc for Debian/Ubuntu,(.srpm for Fedora/RHEL) and upload it to the building repository.
• Does not use ROS tools like colcon, rosdep, bloom. By this time all
package information has been rendered into distribution-specific formats.
• A successful build will trigger binary package builds for all binary
packages generated from the created source package.
• Old binary packages are removed when a new source package isimported.

binary package jobs
Examples Rbin_uJ64__rclcpp__ubuntu_jammy_amd64__binary• Fetch the source from the building repository.
• Installs build dependencies.
• Uploads the binary package created to the building repository.
• Does not use ROS tools like colcon, rosdep, bloom. By this time all
package information has been rendered into distribution-specific formats.
• A successful build will complete by importing the newly built binary
package into the building repository and invalidating all package’s
downstream of it.

Package invalidation• Remove all packages which depend recursively on the newly
rebuilt package.
• Prevents ABI breaks in production.
• Instant feedback for downstream packages when APIschange.

3. Waiting for a sync

sync to testing jobs
Examples Rrel_sync-packages-to-testing_jammy_amd64,
Rrel_sync-packages-to-testing_rhel_8_x86_64• Removes all ROS packages from the testing repository whichmatch the job’s rosdistro, platform version, and architecture,
as well as associated source packages.
• Imports all ROS packages from the building repository whichmatch the same criteria.

sync to testing jobs
• This way, packages are deleted when there is no longer abinary job for them.
• All binary and source jobs block the testing jobs so these run
only once during large rebuilds.
• Automated thresholds are set to prevent a bad release frompurging the testing repository.

sync to main jobs
Examples Frel_sync-packages-main,
Hrel_sync-packages-to-main• Removes all ROS packages from the main
repository which match the job’s rosdistro for eachplatform version and architecture combination.
• Only ever run manually by the ROS core team.
There is a separate subordinate Jenkins job for handling RPM repositories

Thanks!

Modes of integration
Distribution model integration
Software collections are curated and integrated throughcommunity effort. The software available is limited but
using all of it together is much more likely to be
mutually compatible.
Examples: Most Linux distributions (Ubuntu 22.04, Fedora 36, etc),
ROS distributions (Noetic, Humble), Gazebo collections (Citadel,Fortress, Garden), Homebrew.

Modes of integration
Application model integration
Software is aggregated in service of a specificapplication from multiple available sources.
Applications may choose to integrate arbitrary desired
versions of software based on need but each applicationteam must curate their own (often much smaller) set ofdependencies.
Examples: Conda-forge, npm, PyPI, Crates.io/CargoBinary packages
Why do we need them?

Alternatives:
Everyone builds from source’sup Gentoo
Docker / OCICurrently these use the binary packages under
the hood.
Conda / Spack
These are great on platforms without a
distribution packaging system but…