Cowbuilder

Documentation available in: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/package_builder/
== Introduction ==
A module used to populate a Debian/Ubuntu package building environment. Meant to be used in the Wikimedia environment but could be adapted for other environments as well.
== Setting it up ==
Include the package\_builder class in your machine. That can be done with whatever ENC you have puppet working with but in Wikimedia Labs you should create a puppet group, then add the class in the puppet group and just configure your VM with that class
After puppet is done you will have a number of pristine cowbuilder environments. Those would be:
* stretch-amd64
* buster-amd64
* bullseye-amd64
* sid-amd64
== Building packages ==
You just either download a ready package:
 dget http://http.debian.net/debian/pool/main/d/dh-virtualenv/dh-virtualenv_0.10-1.dsc
 export DIST=stretch
 sudo -E cowbuilder --build dh-virtualenv_0.10-1.dsc
or if you are developing a package and are in the package directory:
 DIST=stretch pdebuild
Feel free to change stretch for the distribution of your choice from the ones above
== Architectures ==
ARCH=amd64, or ARCH=i386 is supported if you feel like building for different architecture versions. There is no support for other architectures
== Debugging ==
By default, if the build fails, a hook is executed, providing the user with a shell allowing them to debug the build further. If that's not desired, there's a variable that can be defined to avoid that behavior. Example:
 SHELL_ON_FAILURE=no pdebuild
If you reach the conclusion that your build fails because of some effort to write to HOME, and fixing the software to not do that is unfeasible, then you can set
 BUILD\_HOME = /build
in .pbuilderrc or /etc/pbuilderrc
== Additional Repos ==
=== Wikimedia repos ===
Aside from sid, the rest of the distributions allow for satisfying build time dependencies via the Wikimedia repos.
To use packages from the Wikimedia repos to satisfy build dependencies during building you can use WIKIMEDIA=yes. There is also the approach of appending -wikimedia to the DIST variable and pbuilder will do what you want.
Examples:
$ DIST=stretch-wikimedia pdebuild
$ WIKIMEDIA=yes DIST=stretch pdebuild
The commands above are equivalent and will both build a package for the stretch distribution using the Wikimedia apt repository.
=== Wikimedia Experimental repos ===
We can also add the wikimedia experimental repo using WIKIMEDIA_EXPERIMENTAL=yes in addtion to WIKIMEDIA=yes
$ WIKIMEDIA_EXPERIMENTAL=yes DIST=stretch-wikimedia pdebuild
$ WIKIMEDIA_EXPERIMENTAL=yes WIKIMEDIA=yes DIST=stretch pdebuild
=== Backports repos ===
Packages from the Debian backports repositories can be used to satisfy dependencies as well. To use the backports repository for the distribution selected (e.g. stretch-backports), use either of:
 DIST=stretch BACKPORTS=yes pdebuild
 DIST=stretch-backports pdebuild
=== Archived suites ===
Backports is only available for the regular lifetime of a Debian release, not during the LTS stage. After that packages should ideally be rebuilt using internally built components. If this isn't possible for some reason, the ARCHIVE\_BACKPORTS hook can be used to pull in the backports repository from archive.debian.org.
 DIST=buster ARCHIVE\_BACKPORTS=yes pdebuild
=== Combining Wikimedia and Backports repos ===
Set both WIKIMEDIA and BACKPORTS:
 DIST=stretch BACKPORTS=yes WIKIMEDIA=yes pdebuild
When using a distribution suffix, the other repo must be enabled via an environment variable. The following examples are equivalent:
 DIST=stretch-backports WIKIMEDIA=yes pdebuild
 DIST=stretch-wikimedia BACKPORTS=yes pdebuild
== Additional Components ==
If you need to add a specific WMF component to the build environment then you can use the environment variable COMPONENT, you will also need to set WIKIMEDIA=yes e.g.
$ COMPONENT=component/puppet7 WIKIMEDIA=yes DIST=bullseye pdebuild
 
as well as this we also have a number of environment variables for specific combination of components to allow building common packages components
=== Spicerack ===
use the SPICERACK environment variable
$ SPICERACK=yes pdebuild
=== Cergen ===
use the CERGEN environment variable, you also need to specify DIST=buster as this package only works on buster
$ CERGEN=yes DIST=buster pdebuild
=== Java 8 ===
use the JDK8 environment variable
$ JDK8=yes pdebuild
=== Opensearch 1.0 ===
use the OPENSEARCH1 environment variable
$ OPENSEARCH1=yes pdebuild
=== PHP 7.2 ===
use the PHP72 environment variable
$ PHP72=yes pdebuild
=== PHP 7.4 ===
use the PHP74 environment variable
$ PHP74=yes pdebuild
=== VP 9 ===
use the VP9 environment variable
$ VP9=yes pdebuild
== Using built packages as dependencies ==
By default, cowbuilder will always prefer upstream packages to packages you have built. If you have built a new version of a dependency, you typically want to use that version rather than the one provided by upstream. To force cowbuilder to use packages in the result directory, use:
 APT_USE_BUILT=yes sudo -E cowbuilder --build dh-virtualenv_0.10-1.dsc
=== Adding packages from a local directory ===
The following is useful if you only need a specific package to satisfy the build dependencies. you need to set APT_USE_BUILT=yes and BUILDRESULT pointing to the directory with the deb files e.g.
$ APT_USE_BUILT=yes BUILDRESULT=/home/jbond/debs/build DIST=bullseye pdebuild
== git-buildpackage ==
git-pbuilder can be used by git-buildpackage to leverage all of the above but instead of DIST and ARCH you need to use --git-dist=$DIST and --git-arch=$ARCH.
 WIKIMEDIA=yes gbp buildpackage -sa -us -uc --git-pbuilder --git-no-pbuilder-autoconf --git-dist=stretch
-sa is being used to enforce the original tarball to be included in the .changes file which is a requirement for Wikimedia reprepro.
The --git-no-pbuilder-autoconf/GIT\_PBUILDER\_AUTOCONF=no tells git-pbuilder to forego all attempts to discover the base path, tarball, or configuration file to set up the pbuilder options but rather instead rely on the settings in .pbuilderrc
You can make it a default by editing your ~/.gbp.conf:
 [buildpackage]
 pbuilder = True
 pbuilder-autoconf = False
 dist = buster
Without "dist = buster" gbp will build for sid by default.
== Results ==
The resulting deb files should be in /var/cache/pbuilder/result/${DIST}-${ARCH} like:
 /var/cache/pbuilder/result/stretch-amd64/
== Notes ==
If you are getting confused over the naming of pbuilder/cowbuilder, here's some info to help you. pbuilder is the actual base software, cowbuilder is an extension to allow pbuilder to use COW (copy on write) instead of slow .tar.gz base files. For all intents and purposes this should be transparent to you as cowbuilder is the default pbuilder builder.
== Networking ==
cowbuilder/pbuilder block networking using Linux namespaces. Technically speaking an *unshare* is done in those environments, but the effect is that you can expect networking to not work.
If your package requires internet access to build successfully, it will not work.
First, try to fix the package. Packages in general should not require internet access to be built for a variety of reasons which will not be explained here.
If that is impossible/undesirable, then set
 USENETWORK=yes
in /etc/pbuilderrc or ~/.pbuilderrc can be used to override that behaviour. Make sure that the building host has internet access though, or else your change will not be useful
== Cloud VPS ==
Apply role::labs::lvm::srv to get additional disk space at /srv.
Apply role::package::builder
Category:Package management
Category:SRE Infrastructure Foundations