Updating netboot image with newer kernel

This text describes the necessary steps to update a Debian netboot image with a more recent kernel (e.g. a kernel from backports or a custom kernel). It is written to be generic as possible, in the hope that it's useful for all Debian users (There's a part below which is specific to the production environment of the Wikimedia Foundation, but it can be skipped/adapted to your deployment)
= Backport Buster 5.10 kernel to Buster =
The specific example used here is the integration of the backport of the updated 5.10 kernel for Buster into a netboot image of buster (which is based on 4.19).
* Download the kernel you want to integrate. In this case we're using the Linux 5.10 kernel for Buster (source package name linux-5.10), available from http://security.debian.org/pool/updates/main/l/linux-5.10/ (Ofc you can also build a custom kernel and use that)
* Extract the deb of the kernel package you want to use:
ar x $KERNELDEB
tar xf data.tar.xz (or possibly .bz or .gz)
* Now download the netinst image you want to modify. For Buster this is the netboot.tar.gz from http://ftp.debian.org/debian/dists/buster/main/installer-amd64/current/images/netboot/
* Unpack the existing initrd image you want to modify
mkdir netboot
sudo unmkinitramfs initrd.gz netboot
Copy the content of lib/modules/$KERNELVERSION from the unpacked kernel deb to netboot/lib/modules (the complete directory including the modules.order and modules.buildin files).
* Now rebuild the initrd:
cd netboot
find . 2>/dev/null | cpio --quiet -H newc -o | gzip > ../initrd-new.gz
It's very likely that the netboot image will also need to be updated with firmware, in particular for PXE boots:
For that, download one of the firmware bundles which are available per Debian release: http://cdimage.debian.org/cdimage/unofficial/non-free/firmware/$DISTRO/current/firmware.tar.gz
I.e. for our revised Buster image we need to download http://cdimage.debian.org/cdimage/unofficial/non-free/firmware/buster/current/firmware.tar.gz
It's important to get the right firmware.tar.gz release, as some firmwares are only compatible with specific versions of the Linux kernel.
A Debian initramfs can consist of one or several gzipped CPIO archives, as such we need to convert the firmware tarball to a CPIO archive:
mkdir firmware
tar -C firmware -zxf firmware.tar.gz
pax -x sv4cpio -s'%firmware%/firmware%' -w firmware | gzip -c >firmware.cpio.gz
* Next we need to append this our previously generated initrd:
cat firmware.cpio.gz >> initrd-new.gz
Now that we have an updated initrd, but you'll also need the kernel itself from the kernel deb extracted above: It's in the boot/ directory of the data.tar.xz extracted above and named like vmlinuz-KERNELVERSION-amd64
You need to add this initrd and kernel image in your PXE boot environment.
= WMF-specific instructions for this below: =
The tftpboot environment is distributed via puppet/volatile. Log into puppetmaster1001 and enter the /var/lib/puppet/volatile/tftpboot/ directory. Make a copy of the existing installer environment and e.g. copy it to e.g. installer-buster510.
Now enter the jessie49-installer/debian-installer/amd64 directory and replace linux with the vmlinuz-KERNELVERSION-amd64 file extracted from the kernel and replace the initrd.gz with the one we rebuilt.
Now force a puppet run on the install servers via Cumin:
sudo cumin A:installserver 'run-puppet-agent'
== Updating production point release ==
The process to update the prodcution point release images has been scripted so follow theses simple instuctions but fall back on the notes above if you hit any errors
* from the puppetmasteres run the the following command to built the debian-installer folder
$ sudo update-netboot-image buster 
--2021-03-29 09:20:55-- http://ftp.us.debian.org/debian/dists/buster/main/installer-amd64/current/images/netboot/netboot.tar.gz
Resolving webproxy.eqiad.wmnet (webproxy.eqiad.wmnet)... 2620:0:861:1:208:80:154:32, 208.80.154.32
Connecting to webproxy.eqiad.wmnet (webproxy.eqiad.wmnet)|2620:0:861:1:208:80:154:32|:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 37534443 (36M) [application/x-gzip]
Saving to: ‘netboot.tar.gz’
netboot.tar.gz 100%[========================================================================>] 35.79M 31.9MB/s in 1.1s 
2021-03-29 09:20:56 (31.9 MB/s) - ‘netboot.tar.gz’ saved [37534443/37534443]
--2021-03-29 09:20:56-- http://cdimage.debian.org/cdimage/unofficial/non-free/firmware/buster/current/firmware.tar.gz
Resolving webproxy.eqiad.wmnet (webproxy.eqiad.wmnet)... 2620:0:861:1:208:80:154:32, 208.80.154.32
Connecting to webproxy.eqiad.wmnet (webproxy.eqiad.wmnet)|2620:0:861:1:208:80:154:32|:8080... connected.
Proxy request sent, awaiting response... 302 Found
Location: http://laotzu.ftp.acc.umu.se/cdimage/unofficial/non-free/firmware/buster/current/firmware.tar.gz [following]
--2021-03-29 09:20:56-- http://laotzu.ftp.acc.umu.se/cdimage/unofficial/non-free/firmware/buster/current/firmware.tar.gz
Reusing existing connection to [webproxy.eqiad.wmnet]:8080.
Proxy request sent, awaiting response... 200 OK
Length: 84721761 (81M) [application/x-gzip]
Saving to: ‘firmware.tar.gz’
firmware.tar.gz 100%[========================================================================>] 80.80M 33.0MB/s in 2.5s 
2021-03-29 09:20:59 (33.0 MB/s) - ‘firmware.tar.gz’ saved [84721761/84721761]
The updated netboot environment can be found in /tmp/tmp.LeRoc5mJaL/buster-installer, if everyone looks fine, move it to /var/lib/puppet/volatile/tftpboot
and make sure to remove /tmp/tmp.LeRoc5mJaL
as the output indicates the installer has been built into a temporary folder. Before moving this folder to volatile first make a backup of the current installer
$ mv /var/lib/puppet/volatile/tftpboot/buster-installer{,-10.8}
$ mv /tmp/tmp.LeRoc5mJaL/buster-installer /var/lib/puppet/volatile/tftpboot
Once this has completed, run puppet agent on the installservers 
sudo cumin A:installserver 'run-puppet-agent'
You should then reimage a server to ensure everything runs as expected
__FORCETOC__