Reprepro

Reprepro is a tool for managing APT repositories. 
Reprepro is able to manage multiple repositories for multiple distribution versions and one package pool. It can process updates from an incoming directory, copy package (references) between distribution versions, list all packages and/or package versions available in the repository, etc.
Reprepro maintains an internal database (a .DBM file) of the contents of the repository, which makes it quite fast and efficient.
For more information about Reprepro, consult the resources under #External links.
== Deployment at WMF ==
Reprepro is installed from the Debian package reprepro and is configured using the files in /srv/wikimedia/conf/.
Note that there are two APT repositories available via apt.wikimedia.org, one containing packages that are allowed to be publicly distributed, and one containing packages that the foundation is not allowed to distribute (typically related to a handful of RAID tools packages where we don't have the source). The repository of private packages is available on http://apt.wikimedia.org:8080 with firewalling in place to prevent access from outside the Wikimedia production realm. Packages and configuration for the public repository is available in the /srv/wikimedia directory on apt1002.wikimedia.org, the private repository is contained within /srv/private. 
Reprepro is running on apt1002.wikimedia.org.
== HOWTO ==
Reprepro is typically invoked via the command sudo -i reprepro [...]. By default reprepro will work on the public repository, to add/remove or modify the private repository, please use the private_reprepro wrapper script, to ensure that the correct environment variables are set. E.g.
 sudo -i private_reprepro [...]
This section explains the most commonly needed actions/tasks involving reprepro.
=== Browse packages ===
https://apt-browser.toolforge.org/
=== List all package versions in the repositories ===
For a given package name, use
 reprepro ls PACKAGE_NAME
For example:
# reprepro ls puppet
puppet | 4.8.2-5~trusty1 | trusty-wikimedia | amd64, i386, source
puppet | 4.8.2-5~bpo8+1 | jessie-wikimedia | amd64, i386, source
puppet | 3.8.5-2~bpo8+2 | jessie-wikimedia | amd64, i386, source
puppet | 4.8.2-5 | stretch-wikimedia | amd64, i386, source
This shows that there are two different builds of the same package version in the repositories hardy-wikimedia and lucid-wikimedia. There is clearly no puppet package in the karmic-wikimedia repository.
To see all packages in a given distribution, use
 reprepro list DISTRIBUTION_NAME
To list a package in particular component for a specific distribution, use the following general command
reprepro -C <COMPONENT_NAME> list <DISTRIBUTION_NAME> [<PACKAGE_NAME>] 
The below command will list all packages named varnish-modules in the main component
reprepro -C main list buster-wikimedia varnish-modules
You can as well list all packages contained in a specific component (say component/redis2) as so:
reprepro -C component/redis2 list buster-wikimedia
To find all packages in all repositories, use
 reprepro dumpreferences
or a variant thereof (see reprepro --help or man reprepro).
=== Build packages for import ===
See elsewhere on wikitech for notes on building packages, and Debian_packaging_with_dgit_and_CI for more details of our git-based workflow.
=== Automatically import files from an incoming/ directory ===
Reprepro can automatically import packages from an upload directory, as long as all the package fields are setup correctly with the right distribution and component names. It's also vital that the .changes files are present. When all these conditions are met, and all these files have been uploaded to /srv/wikimedia/incoming (e.g. using dupload), you can use:
user@apt1002:~ $ sudo -i reprepro processincoming default
It uses the rules defined in the file /srv/wikimedia/conf/incoming
If the package is rejected by reprepro because one of the package control fields are wrong, or you want to override them for some other reason, use an override file (see below).
It's best to check whether the /srv/wikimedia/incoming/ directory is empty after using procesincoming, because reprepro should have moved/deleted all imported files. Any remaining files have not been processed.
After importing the packages you likely need to export the packages for them to be available via apt-get:
user@apt1002:~ $ sudo -i reprepro export
=== Importing packages ===
You will need three things:
# Component
#;main:Wikimedia native packages, as well as Debian packages that have had source-modifications.
#;component/name:For custom packages serving a purpose more specific than main (e.g. CI-related package in component/ci, Lilypond builds in component/lilypond, or updated PHP versions in component/php-stable).
#;thirdparty/name: Third-party/external repository objects that we mirror/import. (e.g. tracking the latest updates of HAProxy in thirdparty/haproxy-stable).
# Distribution
## Usually: CODENAME-wikimedia, such as bookworm-wikimedia. This is the distribution that the package has been compiled for, and under. This should match the field in the package's Changelog.
## If your package was specifically built for Wikimedia and does not have a distribution of CODENAME-wikimedia listed in the changes file, then you should force reprepro to accept a CODENAME-wikimedia distribution. Add --ignore=wrongdistribution flag to the reprepro command to do so.
# Changes file
## If building on the packaging server, the changes file is most likely in /var/cache/pbuilder/result/CODENAME-amd64/ along with the dsc, buildinfo, tar.xz, deb, and orig.tar.gz.
## You will want to bring all of these files over to the reprepro server.
It's best to have reprepro fully manage all package aspects using the changes file that was created during the build of the package (e.g. using Pbuilder). When the changes file is present along with all files list therein, reprepro can handle it all with:
 # reprepro -C COMPONENT include DISTRIBUTION CHANGES_FILE
For example:
 # reprepro -C main include bookworm-wikimedia somepackage_1.2.3-1+deb12u1+wmf1_amd64.changes
When no changes file is available, for example because you didn't build the package yourself, you can use includedsc and includedeb:
 # reprepro -C main includedsc bookworm-wikimedia somepackage_1.2.3-1.dsc
 # reprepro -C main includedeb bookworm-wikimedia somepackage_1.2.3-1_amd64.deb
Be aware that reprepro will remove older versions of packages without asking. They are no longer available in the pool (/srv/wikimedia/pool) either. However, /srv/wikimedia/ is backed up using Bacula.
=== Missing orig tarball ===
The original tarball should be listed in the changes file when package is built. Often times, this is accomplished with a build flag: dpkg-buildpackage -sa, debuild -sa, or pdebuild -- --debbuildopts -sa
If it is missing though, you can tell reprepro to ignore it and find it in the current working directory with the --ignore=missingfile flag.
=== Removing packages ===
A given binary package can be removed from a distribution (including all of its components, not just main) using
 # reprepro remove distribution-name package-name
For example:
 # reprepro remove jessie-wikimedia facter
However, usually you want to remove all binary packages from a source package, that can be done with. Be careful with that command, the current version of reprepro we use, will remove the package from all disitributions, not just the one specified! (Should be fixed in the version/fork used in bookworm)
 # reprepro removesrc distribution-name package-name
For example:
 # reprepro removesrc trusty-wikimedia openjdk-8
Removing a package from a specific component:
 # reprepro -C thirdparty/elastic65 remove stretch-wikimedia kibana
=== Removing a component ===
After removing a component's configuration in Puppet, the packages and directories on apt1002 still need to be cleaned out by hand:
root@apt1002:/srv/wikimedia# reprepro clearvanished 
There are still packages in 'buster-wikimedia|thirdparty/amd-rocm25|amd64', not removing (give --delete to do so)!
Deleting vanished identifier 'buster-wikimedia|thirdparty/amd-rocm25|i386'.
Deleting vanished identifier 'buster-wikimedia|thirdparty/amd-rocm25|source'.
As indicated above, there likely still will be packages in the tree. You can remove them by using --delete:
root@apt1002:/srv/wikimedia# reprepro --delete clearvanished 
Deleting vanished identifier 'buster-wikimedia|thirdparty/amd-rocm25|amd64'.
Deleting files no longer referenced...
=== Using the override file ===
When we are building our own packages, we should make sure that all control fields (such as the distribution name, component, priority etc.) are set correctly. Please rebuild your package if not.
However, occasionally it is necessary to override fields on a previously built package, which we don't want to modify the source of and/or rebuild. Ubuntu often does the same, and just retrieves packages from Debian Unstable and overrides a few fields using an override file.
We have an override file as well, in /srv/wikimedia/conf/deb-override. It's format is:
 # packagename	fieldname	newvalue
As an example, our Varnish packages are coming straight from Debian Unstable (like in Ubuntu), and can be imported fine into Lucid as long as we override some package fields:
varnish		Distribution	lucid
libvarnish1	Distribution	lucid
libvarnish-dev	Distribution	lucid
=== Copying between distributions ===
In some cases it's necessary to copy a binary which gets reused in a different distribution, e.g. for Go packages (which are statically linked and usually hard/impossible to rebuild in older distro versions). Note that in contrast to typical conventions you first need to specify the destination and then the source!. In the following example we copy from stretch to buster:
# reprepro [-C component/...] copy buster-wikimedia stretch-wikimedia prometheus-rsyslog-exporter
Exporting indices...
# reprepro ls prometheus-rsyslog-exporter
prometheus-rsyslog-exporter | 0.0.0+git20180118-1 | jessie-wikimedia | amd64, source
prometheus-rsyslog-exporter | 0.0.0+git20180118-1 | stretch-wikimedia | amd64, source
prometheus-rsyslog-exporter | 0.0.0+git20180118-1 | buster-wikimedia | amd64, source
If you need to copy all binary packages of a source package, you can use copysrc (and give the source package name) instead.
You can search for the .deb package and re-include it in the repository, in the right component:
user@apt1002:~ $ sudo find /srv/wikimedia/ -name *python-mwclient*
/srv/wikimedia/pool/thirdparty/m/mwclient/python-mwclient_0.8.4-1_all.deb
user@apt1002:~ $ sudo -i reprepro -C <new_component> includedeb <new_repo> /srv/wikimedia/pool/thirdparty/m/mwclient/python-mwclient_0.8.4-1_all.deb
Anyway, this alternative procedure is not common, and you should wonder why the source package is not available.
=== Updating external repositories ===
Reprepro has the ability to pull packages from other APT repositories automatically, this has added benefits like verifying signatures, easy management and so on. It is configured via conf/updates configuration file managed via Puppet.
To check which updates are available:
 # reprepro --component thirdparty/tor checkupdate stretch-wikimedia
Calculating packages to get...
Updates needed for 'stretch-wikimedia|thirdparty/tor|source':
'tor': '0.3.3.9-1~d90.stretch+1' will be upgraded to '0.3.5.7-1~d90.stretch+1' (from 'tor-stretch'):
 files needed: pool/thirdparty/tor/t/tor/tor_0.3.5.7-1~d90.stretch+1.dsc pool/thirdparty/tor/t/tor/tor_0.3.5.7.orig.tar.gz pool/thirdparty/tor/t/tor/tor_0.3.5.7-1~d90.stretch+1.diff.gz
 Updates needed for 'stretch-wikimedia|thirdparty/tor|amd64':
'tor': '0.3.3.9-1~d90.stretch+1' will be upgraded to '0.3.5.7-1~d90.stretch+1' (from 'tor-stretch'):
 files needed: pool/thirdparty/tor/t/tor/tor_0.3.5.7-1~d90.stretch+1_amd64.deb
'tor-dbgsym': '0.3.3.9-1~d90.stretch+1' will be upgraded to '0.3.5.7-1~d90.stretch+1' (from 'tor-stretch'):
 files needed: pool/thirdparty/tor/t/tor/tor-dbgsym_0.3.5.7-1~d90.stretch+1_amd64.deb
'tor-geoipdb': '0.3.3.9-1~d90.stretch+1' will be upgraded to '0.3.5.7-1~d90.stretch+1' (from 'tor-stretch'):
 files needed: pool/thirdparty/tor/t/tor/tor-geoipdb_0.3.5.7-1~d90.stretch+1_all.deb
You can also get an overview of all packages currently pending for a distro using 'checkupdate':
reprepro checkupdate stretch-wikimedia
Similarly, you can restrict which packages are affected by update/checkupdate with:
reprepro --noskipold --restrict grafana checkupdate
To pull in the updates for real:
 # reprepro --noskipold --component thirdparty/php72 update stretch-wikimedia
Calculating packages to get...
Getting packages...
Installing (and possibly deleting) packages...
Exporting indices...
Deleting files no longer referenced...
Sometimes a repository might contain more than one version of a package. In such a case you can also specifically target a given version with e.g.
reprepro -C thirdparty/ci --noskipold --restrict-binary=jenkins=2.100-2 checkupdate buster-wikimedia
=== Adding a new external repository ===
When adding a new external repository, you need to: 
# add a new definition to modules/aptrepo/files/updates in puppet.git 
# add the repo's ASCII public key(s) in modules/aptrepo/files/updates-keys/<LONG_KEYID>_<name>.gpg
# add the new component to aptrepo/files/distributions-wikimedia
See 675812 for an example CR. Once the CR is merged run the following command to preform the initial sync
$ sudo -i reprepro --verbose --component thirdparty/gitlab update buster-wikimedia 
.....snip.....
aptmethod redirects 'https://packages.gitlab.com/gitlab/gitlab-ce/debian/dists/buster/main/binary-amd64/Packages.bz2' to 'https://d20rj4el6vkp4c.cloudfront.net/7/8/debian/dists/buster/main/binary-amd64/Packages.bz2?t=1618409760_6b07d0289d193c0fe2b28b704d6a5efe45eeb096'
aptmethod got 'https://d20rj4el6vkp4c.cloudfront.net/7/8/debian/dists/buster/main/binary-amd64/Packages.bz2?t=1618409760_6b07d0289d193c0fe2b28b704d6a5efe45eeb096'
Calculating packages to get...
Getting packages...
aptmethod redirects 'https://packages.gitlab.com/gitlab/gitlab-ce/debian/pool/buster/main/g/gitlab-ce/gitlab-ce_13.10.3-ce.0_amd64.deb' to 'https://d20rj4el6vkp4c.cloudfront.net/7/8/debian/package_files/97310.deb?t=1618409761_57fc9bd551c3fae0715cb4ab072ac304dcdf8d4b'
aptmethod got 'https://d20rj4el6vkp4c.cloudfront.net/7/8/debian/package_files/97310.deb?t=1618409761_57fc9bd551c3fae0715cb4ab072ac304dcdf8d4b'
Then get the key ID:
$ curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.24/deb/Release.key | gpg 2>/dev/null | grep "^ " | tr -d ' ' | tail -c 17
234654DA9A296436
=== Updating where Pull rules are used ===
If a Pull rule is configured between internal distributions, the checkpull and pull commands needs to be used instead to copy packages. For example, thirdparty/cloudera is pulled from an external repository to jessie-wikimedia and then a Pull rule is present for stretch-wikimedia, so an upgrade should follow this procedure: Analytics/Systems/Cluster/Hadoop/Administration#Updating Cloudera Packages.
=== If signing fails ===
If you get errors such as: 
 Error: gpgme created no signature!
 This most likely means gpg is confused or produces some error libgpgme is
 not able to understand.
Make sure you are running as root AND you have exported its environment variables to point to the right gpg keyring (at least until other method of signing is setup). This usually means:
 sudo -i
or, better,
 export REPREPRO_BASE_DIR=/srv/wikimedia
 export GNUPGHOME=/root/.gnupg
 sudo -E ...
=== wrong reprepro base dir ===
Beware, reprepro is installed in different places, on apt* servers but also on releases* servers and they do not use the same reprepro base dir path.
Optionally you can put something like this in your .bash_profile:
 16 # set the right base dir for reprepro, depending whether it's apt.wm.org or releases.wm.org
 17 if [ "$(hostname -s | cut -c 1-8)" == "releases" ]; then
 18 export REPREPRO_BASE_DIR=/srv/org/wikimedia/reprepro
 19 fi
 20 if [ "$(hostname -s | cut -c 1-3)" == "apt" ]; then
 21 export REPREPRO_BASE_DIR=/srv/wikimedia
 22 fi
You could also export the GNUPGHOME or other needed things there. Your .bash_profile can be puppetized in the operations/puppet repo under modules/admin/files/home.
=== Multiple versions of the same package ===
If you need to keep multiple versions of the same package in the repo, you will need to create versioned components, as reprepro doesn't support keeping foo_10-1.deb and foo_20-1.deb at the same time in the same component.
There should be plenty of examples on how to do this in the puppet repository.
TODO: perhaps add here some concrete examples for documentation purposes.
== Additional info ==
Some additional bits you may find useful.
=== dput ===
You can upload packages to the incoming directory using dput too. Create a config file ~/dput.wmf.cf:
[wmf]
fqdn = apt1002.wikimedia.org
login = myuser
incoming = /srv/wikimedia/incoming
method = scp
And then use the tool like this:
user@laptop:~/pkg$ dput -c ~/dput.wmf.cf wmf ../pyasn1_0.4.2-3~bpo9+1~wmf1_amd64.changes 
Checking signature on .changes
gpg: ../pyasn1_0.4.2-3~bpo9+1~wmf1_amd64.changes: Valid signature from 68E713981D1515F8
Uploading to wmf (via scp to apt1002.wikimedia.org):
pyasn1_0.4.2-3~bpo9+1~wmf1_amd64.buildinfo 100% 8146 67.9KB/s 00:00 
pypy-pyasn1_0.4.2-3~bpo9+1~wmf1_all.deb 100% 56KB 137.4KB/s 00:00 
python-pyasn1-doc_0.4.2-3~bpo9+1~wmf1_all.deb 100% 112KB 181.5KB/s 00:00 
python-pyasn1_0.4.2-3~bpo9+1~wmf1_all.deb 100% 56KB 93.2KB/s 00:00 
python3-pyasn1_0.4.2-3~bpo9+1~wmf1_all.deb 100% 56KB 137.7KB/s 00:00 
pyasn1_0.4.2-3~bpo9+1~wmf1_amd64.changes 100% 3166 15.0KB/s 00:00 
Successfully uploaded packages.
=== dupload ===
To upload .changes files with dupload you will likely need a configuration like this in /etc/dupload.conf:
[...]
$cfg{'wmf'} = {
 fqdn => 'apt1002.wikimedia.org',
 method => 'scp',
 incoming => '/srv/wikimedia/incoming',
 login => 'yourusername',
};
[...]
And then, you will be able to upload to incoming using a command like:
$ dupload --to wmf ../pkg_version_amd64.changes
=== testing update filters ===
If you are working on a filter for modules/aptrepo/files/updates, you can develop/test such filter locally before pushing the change to reprepro and finding out it doesn't work as expected.
Basically, download to your laptop a Packages file and use the grep-dctrl tool locally to see the filtered/generated file resulting of applying the filter.
user@debian:~$ wget https://packages.cloud.google.com/apt/dists/kubernetes-xenial/main/binary-amd64/Packages
[..]
user@debian:~$ grep-dctrl \( -P 'kubeadm' -o -P 'kubelet' -o -P 'kubectl' -a -FVersion --lt 1.16 -a -FVersion --ge 1.15 \) -o \( -P 'kubernetes-cni' -o -P 'cri-tools' \) < Packages | less
[..]
== External links ==
* official manual
* reprepro(1) manpage
* Debian Reprepro configuration tutorial.
Category:Ubuntu
Category:Package management
Category:SRE Infrastructure Foundations