Portal:Toolforge/Admin/Packaging

This page contains information on Debian packages (.deb) related to Toolforge.
== Debian Packages ==
Some Toolforge components (e.g. Toolforge CLIs) are deployed using Debian Packages. Each component is packaged separately, the .deb file is uploaded to the a Debian repository running in tools-services-* and from there it is finally installed on the Toolforge bastions.
This is a list of the Debian packages installed on the bastions as of April 2024 (please note this list may change in the future):
* toolforge-cli: https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli
* toolforge-builds-cli: https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli
* toolforge-envvars-cli: https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli
* toolforge-jobs-framework-cli: https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli
* toolforge-webservice: https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice
* misctools: https://gerrit.wikimedia.org/r/admin/projects/labs/toollabs
Note: When the tools-webservice package is updated, all docker images need to be rebuilt.
== Deploy process ==
* Bump version: Before building a new version of a package.
* Build package
* Distribute and deploy package
== Using existing automation ==
=== Bump the version ===
All components provide a utils/bump_version.sh. It will take care of everything:
* creating the bump branch
* updating the debian/changelog file
* updating the version in setup.py or in other files (depending on the component)
* creating a new git tag locally
If the script completes successfully, review the changes with git show and if they look good create a merge request.
The package will be built automatically for you on CI, you can deploy it in lima-kilo with toolforge_deploy_mr.py for testing.
Once the MR is approved, don't merge it yet, we will deploy the package.
=== Deploy the package ===
Before merging the version bump branch, you can deploy the package from that branch to tools and toolsbeta, allowing you to test before finally merging.
You can do so by using the following cookbook:
root@cloudcumin1001$ cookbook wmcs.toolforge.component.deploy --cluster-name tools --component jobs-cli
Got matching cookbooks wmcs.toolforge.component.deploy
START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli
INFO: Uploading artifacts to tools-services-06.tools.eqiad1.wikimedia.cloud at /tmp/cookbook-toolforge-k8s-component-deploy-nissklpbsg (same node for tools and toolsbeta)
INFO: Publishing artifacts on repo buster-tools
INFO: Publishing artifacts on repo bullseye-tools
INFO: Publishing artifacts on repo bookworm-tools
INFO: cleaning up temp dir /tmp/cookbook-toolforge-k8s-component-deploy-nissklpbsg
INFO: Installing packages on all bastions for project tools
INFO: installed toolforge-jobs-framework-cli version 16.1.0 on tools-bastion-[12-13].tools.eqiad1.wikimedia.cloud,tools-sgebastion-10.tools.eqiad1.wikimedia.cloud
... tests run
Stopping proxy on 127.0.0.1:12345
END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli
If the cookbook does not work or you want to deploy a specific manually build package, see the #Manually section below.
=== Merge the MR and create the release ===
After the package has been deployed and tested, you can merge the MR in GitLab, and push the git tag with git push origin {tagname}.
Then create a new release in GitLab by clicking on "Releases", then on "New release".
== Manually ==
=== Bumping the version ===
Before building a new version of a package, you need to bump the package version.
If the bump_version script is not present or is not working correctly, you can bump the version manually:
* To update the debian/changelog file, you can use: gbp dch --git-author --id-length=7 --since=... . This reads the git log to generate changelog entries.
** Not on a Debian host? You can run gbp inside Docker: 
docker run -it --rm \
 -v "$(pwd):/mnt" -v "$HOME/.gitconfig:/etc/skel/.gitconfig:ro" \
 -e UID=$(id -u) -e GID=$(id -g) \
 opxhub/gbp:buster \
 bash -c 'gbp dch --git-author --id-length=7'
**The Docker trick can also be included in a bash init script as a function for your shell so you only need to run gbp and then the rest of the command exactly like on a Debian host 
function gbp() {
 docker run -it --rm -v "$(pwd):/mnt" \
 -v "$HOME/.gitconfig:/etc/skel/.gitconfig:ro" -e UID="$(id -u)" \
 -e GID="$(id -g)" opxhub/gbp:buster bash -c "gbp $*"
}
* Update the version in setup.py and in other files where is present (depending on the component)
* Create a new tag with the format debian/version. Optionally you can create an annotated tag (git tag -a tagname) and add a short annotation, e.g. the output of git shortlog {previoustag}..HEAD
=== Building the package ===
Most packages have a CI pipeline that will build the .deb file automatically. Click on the pipeline and look for a job named "package:deb". The .deb file can be downloaded clicking on "Download artifacts".
If the component does not have a CI pipeline, check if it includes a file utils/build_deb.sh. Otherwise, you can use the wmcs-package-build script (more details below).
=== Uploading a package ===
If you built a package or have a package already built that you want to upload, you can scp it to tools-services-* VM, and from there add it to the repository you want it in:
$ scp pack-0.27.0-2.deb tools-services-06.tools.eqiad1.wikimedia.cloud:/tmp
$ ssh tools-services-06.tools.eqiad1.wikimedia.cloud
...
# check which repositories you want to add the package to
root@tools-services-06:~# aptly repo list
...
# Add the package to the repo you want
root@tools-services-06:~# aptly repo add buster-toolsbeta /tmp/pack-0.27.0-2.deb
...
# Publish the package on each of the repos where you added it
root@tools-services-06:~# aptly publish update --skip-signing bullseye-toolsbeta
...
You should always install a package in the toolsbeta project first and test it works correctly, before installing it in the tools project.
For that, build the package and upload it to the -toolsbeta variants of the aptly repos, e.g. bullseye-toolsbeta.
Once testing is done and you are confident with a given package version, you can copy the packages to the -tools repos:
root@tools-services-06:~# aptly repo copy buster-toolsbeta buster-tools toolforge-webservice_0.XX_all && aptly publish --skip-signing update buster-tools
root@tools-services-06:~# aptly repo copy bullseye-toolsbeta bullseye-tools toolforge-webservice_0.XX_all && aptly publish --skip-signing update bullseye-tools
=== Installing a package ===
Packages will not be upgraded automatically by Puppet. You need to manually run apt update && apt install {package_name}.
You can use Cumin to push the change out everywhere:
$ ssh cloudcumin1001.eqiad.wmnet
$ sudo cumin "O{project:tools name:.*-bastion.*}" "apt-get update"
$ sudo cumin "O{project:tools name:.*-bastion.*}" 'DEBIAN_FRONTEND=noninteractive apt-get install -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" toolforge-webservice'
== Obsolete methods ==
=== wmcs-package-build ===
This is a custom script to automate most of the build & publish steps of the process. You can locate this script in the operations/puppet repo: modules/toolforge/files/wmcs-package-build.py.
The steps are done in order:
* git stage: create a random directory in the package build server, git clone the source repository and checkout a given branch.
* sbuild stage: run sbuild to build the debian package.
* copy stage: copy the resulting .deb packages from the package build server into the aptly server at a random directory.
* aptly stage: add the .deb packages to a given repo and publish it.
* backup stage: run rsync from the aptly tree to NFS (optional).
* cleanup stage: delete random directories created by the script.
user@laptop:~$ modules/toolforge/files/wmcs-package-build.py --help
usage: wmcs-package-build.py [-h] --git-repo GIT_REPO [--git-branch GIT_BRANCH]
 [--build-dist BUILD_DIST] [--build-host BUILD_HOST] [-a APTLY_DIST]
 [--aptly-host APTLY_HOST] [-b] [-d]
Utility to build and upload a .deb package to aptly
optional arguments:
 -h, --help show this help message and exit
 --git-repo GIT_REPO git repository URL with the source pkg. This script will do a fresh git clone
 of that repo. Typical value is something like:
 https://gerrit.wikimedia.org/r/operations/software/tools-webservice
 --git-branch GIT_BRANCH
 git branch to use to build the package from. Defaults to "master"
 --build-dist BUILD_DIST
 target distribution when building the package with sbuild. Defaults to
 "stretch"
 --build-host BUILD_HOST
 package build host. Typically a VM in CloudVPS with
 role::wmcs::toolforge::package_builder. Defaults to "tools-package-
 builder-02.tools.eqiad.wmflabs"
 -a APTLY_DIST, --aptly-dist APTLY_DIST
 target distribution in aptly. The resulting deb package will be uploaded to
 this distribution and then the repository will be published. Can be specified
 multiple times for multiple target distributions. If this argument is not
 provided, no aptly operations will be done. Example: -a stretch-tools -a
 stretch-toolsbeta
 --aptly-host APTLY_HOST
 aptly server host. Typically a VM in CloudVPS with
 role::wmcs::toolforge::services. Defaults to "tools-sge-
 services-03.tools.eqiad.wmflabs"
 -b, --no-backup If this option is present, this script won't backup aptly data over NFS
 -d, --dry-run Dry run: only show what this script would do, but don't do it for real
This script is supposed to run from your laptop, and it will start SSH connections to the different servers involved in building and publishing a deb package.
The --help output should have enough information to know what every option does.
You are encouraged to use the --dry-run first to know what the script would do (the actual SSH commands).
=== sbuild ===
The sbuild tool is a simple and powerful tool to build debian packages.
Our packaging building hosts have sbuild ready to use. The first time you try to use sbuild you should add yourself to the sbuild unix group:
user@tools-package-builder-02:~$ sudo sbuild-adduser $LOGNAME
[..]
user@tools-package-builder-02:~$ newgrp sbuild
There should be environments to build packages for the most common distros: jessie, stretch, buster.
You can switch between them easily:
user@tools-package-builder-02:mypackage/$ sbuild -d jessie
[..]
user@tools-package-builder-02:mypackage/$ sbuild -d stretch
[..]
user@tools-package-builder-02:mypackage/$ sbuild -d buster
[..]
Sometimes the build-deps required to clean the source tree are not installed outside the chroot. To workardound this, use the --no-clean-source switch when calling sbuild.
Adding extra repos and packages is super easy, check the --extra-repository option in the sbuild manpage.
Additional upstream documentation: https://wiki.debian.org/sbuild
=== Yuvi's method (debuild) ===
# ssh to tools-package-builder-02.tools.eqiad1.wikimedia.cloud
# cd into your clone of the toollabs repo
# check out the branch master
# git pull
# debuild -uc -us -I
# ssh to tools-sge-services-03.tools.eqiad1.wikimedia.cloud
# Add the packages to aptly
## sudo aptly repo add jessie-tools jobutils_$VERSION_all.deb
## sudo aptly publish --skip-signing update jessie-tools
## sudo aptly repo add stretch-tools jobutils_$VERSION_all.deb
## sudo aptly publish --skip-signing update stretch-tools
# Back up the repositories with sudo rsync --chmod 440 --chown root:"${INSTANCEPROJECT}".admin -ilrt /srv/packages/ /data/project/.system/aptly/"$(hostname -f)" (-n for dry run)
# Run sudo -i apt-get update && sudo -i apt-get install jobutils on tools-bastion-02
# Test jsub on tools-bastion-02 to make sure the package works.
# Update the package on the rest of the cluster:
$ ssh tools-clushmaster-02.tools.eqiad1.wikimedia.cloud
$ clush -w @bastion,@cron -b '/usr/bin/dpkg -s jobutils &>/dev/null && sudo apt-get update -qq && sudo /usr/bin/env DEBIAN_FRONTEND=noninteractive apt-get install -q -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" jobutils'
=== Bd808's method (pdebuild) ===
# ssh tools-package-builder-02.tools.eqiad1.wikimedia.cloud
# cd /srv/src/toollabs
# git checkout master
# git pull
# DIST=jessie pdebuild
# export VERSION=$(dpkg-parsechangelog --show-field version)
# echo $VERSION
# cp /srv/pbuilder/result/jessie-amd64/jobutils_${VERSION}_all.deb ~
# ssh tools-sge-services-03.tools.eqiad1.wikimedia.cloud
# sudo aptly repo add jessie-tools jobutils_${VERSION}_all.deb
# sudo aptly publish --skip-signing update jessie-tools
# sudo aptly repo add stretch-tools jobutils_${VERSION}_all.deb
# sudo aptly publish --skip-signing update stretch-tools
# Back up the repositories with sudo rsync --chmod 440 --chown root:"${INSTANCEPROJECT}".admin -ilrt /srv/packages/ /data/project/.system/aptly/"$(hostname -f)" (-n for dry run)
# Run sudo -i apt-get update && sudo -i apt-get install jobutils on tools-bastion-02
# Test jsub on tools-bastion-02 to make sure the package works.
# Update the package on the rest of the cluster:
$ ssh tools-clushmaster-02.tools.eqiad1.wikimedia.cloud
$ clush -w @all -b '/usr/bin/dpkg -s jobutils &>/dev/null && sudo apt-get update -qq && sudo /usr/bin/env DEBIAN_FRONTEND=noninteractive apt-get install -q -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" jobutils'
== See also ==
* phabricator T249837 - Revamp the build process for debian packages in Toolforge -- task that originated the wmcs-package-build.py script.
* phabricator T272290 - wmcs-package-build.py: add support for creating a git tag -- future improvement to the packaging workflow.
* phabricator T272289 - wmcs-package-build.py: add support for testing the packages -- another future improvement to the packaging workflow.