Debian Glue

Debian Glue is a project to facilitate easier Debian package building in Jenkins jobs. Debian Glue utilizes cowbuilder, which itself is a tool for building Debian packages inside a clean chroot. The jobs use a series of shell wrappers which streamlines the environment setup and Debian toolchain.
The jobs are running on a dedicated Jenkins agents hosted on WMCS. They come with cowbuilder images provisioned by our Puppet module package_builder which:
* creates images for the distribution we care about (ex: trusty, jessie, unstable)
* auto updates the images on a daily basis
* provides hooks to support injecting apt.wikimedia.org components and the <release>-backports distribution (example: buster-backports).
For Gerrit repositories having Debian packages, the Jenkins job is registered in a Zuul workflow to trigger:
; integration/config.git /zuul/layout.yaml
 - name: operations/debs/contenttranslation/apertium
 test:
 - 'debian-glue'
It instructs CI to run the debian-glue job for any patchset proposed to that Gerrit repository. But it only triggers if the change touches a file under the debian directory. The job then:
* clones the repository
* checks out the patch that has been merged by CI against the tip of the targeted branch
* sets distribution to the distribution mentioned in debian/changelog (see [#Distributions] below).
* invokes Jenkins Debian Glue.
Jenkins Debian Glue will set DIST based on distribution. Then it triggers the build process in the matching cowbuilder image. When pbuilder is run, the Wikimedia hooks are invoked and whenever the debian/changelog distribution is suffixed with -wikimedia, more specific components (eg: thirdparty) will be added.
Specific environment variables can be injected by Zuul. This is done by altering a Python script executed by Zuul whenever it triggers a job. That can be used to set BACKPORTS which instruct our hooks to inject the release-backports components, tweak the build timeout with BUILD_TIMEOUT or pass DEB_BUILD_OPTIONS. Example as of May 2020:
; integration/config.git /zuul/parameter_functions.py
 if 'debian-glue' in job.name:
 # XXX
 # When adding new paramters, make sure the env variable is added as an
 # env_keep in the sudo policy:
 # https://horizon.wikimedia.org/project/sudo/
 #
 if 'nocheck' in job.name:
 params['DEB_BUILD_OPTIONS'] = 'nocheck'
 if 'backports' in job.name: # T173999
 params['BACKPORTS'] = 'yes'
 # Always set the value to be safe (T144094)
 params['BUILD_TIMEOUT'] = 30 # minutes
 # Finely tweak jenkins-debian-glue parameters
 if params['ZUUL_PROJECT'] == 'integration/zuul':
 # Uses dh_virtualenv which needs access to pypy.python.org
 params['PBUILDER_USENETWORK'] = 'yes'
 elif (params['ZUUL_PROJECT'] == 'operations/debs/varnish4'):
 # VTC tests take forever 
 params['BUILD_TIMEOUT'] = 60 # minutes
 params['DEB_BUILD_OPTIONS'] = 'parallel=12'
 elif (params['ZUUL_PROJECT']
 == 'operations/software/varnish/varnishkafka'):
 # needed for librdkafka1 >= 0.11.5
 params['BACKPORTS'] = 'yes'
 elif (params['ZUUL_PROJECT'] == 'operations/software/atskafka'):
 # needed by go build to access gopkg.in
 params['PBUILDER_USENETWORK'] = 'yes'
 elif (params['ZUUL_PROJECT'] == 'operations/debs/trafficserver'):
 # Building ATS takes a while
 params['BUILD_TIMEOUT'] = 60 # minutes
 # Backports needed on stretch for libbrotli-dev and a recent
 # debhelper version (>= 11)
 params['BACKPORTS'] = 'yes'
 elif (params['ZUUL_PROJECT']
 == 'operations/debs/contenttranslation/giella-sme'):
 # Heavy build T143546
 params['BUILD_TIMEOUT'] = 180 # minutes
Since the build is done with sudo cowbuilder, each new environment variable has to be whitelisted in the sudo policy of the integration labs project.
Non-exhaustive list as of September 2016:
Env 	Description
DEB_* 	???
DIST 	Distribution used by cowbuilder, pbuilder and Wikimedia pbuilder hook
ARCH 	Architecture (i386, amd64)
BUILDRESULT 	Debian glue setting
distribution 	Debian glue setting. Set by the job to the distribution in debian/changelog
WORKSPACE 	Base directory of the Jenkins job. Set by Jenkins.
DEB_BUILD_OPTIONS	 Defined by Debian Policy, let you change behavior of the build process if proper support is added in debian/rules. A typical example is to bypass tests: DEB_BUILD_OPTIONS=nocheck. If needed, must be injected by Zuul.
== Distributions ==
Jenkins debian glue parses the debian/changelog to find the target distribution. When the changelog entry targets UNRELEASED, CI picks the distribution from the previous changelog entry.
In some case ones want to set both WIKIMEDIA and BACKPORTS, but both cannot be set in the changelog entry. Our convention is to suffix the distribution with -wikimedia in the changelog and then trigger the more specific job debian-glue-backports which causes CI to set BACKPORT=yes.
Some repositories want a single branch to support multiple distributions. We thus have a set of jobs that hardcode the distribution and do not extract it from debian/changelog. Examples:
* debian-glue-stretch
* debian-glue-buster
* debian-glue-unstable
This is an example for labs/toollabs which resides in </code>zuul/layout.yaml</code> has:
 - name: labs/toollabs
 test:
 # Single branch supporting multiple distributions T210780
 - debian-glue-unstable
 - debian-glue-stretch
 - debian-glue-buster
 gate-and-submit:
 - debian-glue-unstable
 - debian-glue-stretch
 - debian-glue-buster
== References ==
* Project Homepage
* Jenkins Debian Glue settings with documentation: http://jenkins-debian-glue.org/docs/
* Zuul workflow https://phabricator.wikimedia.org/diffusion/CICF/browse/master/zuul/layout.yaml
* Zuul python hook to inject env variables https://phabricator.wikimedia.org/diffusion/CICF/browse/master/zuul/parameter_functions.py
* Wikitech sudo policy: Special:NovaSudoer
* Definition of our Jenkins jobs using the Jenkins Job Builder DSL: https://phabricator.wikimedia.org/diffusion/CICF/browse/master/jjb/operations-debs.yaml