Modprobed-db

modprobed-db
is a utility that populates a list of all the
kernel modules
that have been loaded on a system while running. This list can then be used to disable all the unused modules when
building your own kernel
and significantly reduce the compilation time.
Installation
Install
the
modprobed-db
AUR
package.
Run
modprobed-db
to create
$XDG_CONFIG_HOME/modprobed-db.conf
if it does not already exist.
Run
modprobed-db store
to create the
$XDG_CONFIG_HOME/modprobed.db
database file and populate it with the currently loaded modules.
Optionally:
add modules in the ignore array that you do *not* want counted, for example modules that get built or that are provided by another package.
Some common ones are included by default:
$ cat ~/.config/modprobed-db.conf
IGNORE=(nvidia vboxdrv vboxnetflt vboxnetadp vboxpci lirc_dev lirc_i2c
osscore oss_hdaudio oss_usb tp_smapi thinkpad_ec
zavl znvpair zunicode zcommon zpios zfs spl splat)
Usage
Populating the database
Once the initial database has been created, use
modprobed-db list
to show the current database modules and
modprobed-db store
to update the database with the currently loaded kernel modules.
Recommendations
It is recommended to "use" the official Arch kernel, with
modprobed-db
installed, for a good amount of time to allow the database to grow based on usage and capture everything the system needs before
building a kernel without the unneeded modules
). Here are some suggested actions to allow appropriate modules to load and get cataloged:
Mount every kind of
filesystem
that will be used.
Insert every kind of removable media that will be used (USB, DVD, CD, etc.).
This includes the mounting of ISO image files if applicable, which uses the
loop
and
isofs
modules.
Use every device on the machine (network interfaces, input devices, webcams, mobile devices, etc.).
Use every desired application, as some depend on kernel modules. For example:
IP blocking/filtering software such as
pgl-cli
AUR
require the
iptables
kernel module.
Encryption software such as
veracrypt
requires cryptography kernel modules. Be sure to mount some encrypted containers so they will actually be loaded.
Certain
QEMU
configurations require kernel modules.
lm-sensors
requires kernel modules for reading hardware sensors.
Try re-running modprobed-db while running different Linux-kernels; they may include modules not enabled in the other kernels.
These suggested actions are to be made in parallel with periodically updating the database with
modprobed-db store
to capture any newly loaded kernel module.
Automatic periodic database updates
Calls to
modprobed-db store
can be automated with one of the following methods:
Cron
The most convenient method to use modprobed-db is to simply add a crontab entry invoking
/usr/bin/modprobed-db store
at some regular interval.
Example running the script once every hour:
$ crontab -e
0 */1 * * *   /usr/bin/modprobed-db store &> /dev/null
systemd
Instead of cron,
enable/start
the
modprobed-db.service
user unit
. It will run modprobed-db in store mode once per 6 hours, and at boot and on shutdown.
Like any service and timer, the
status
of the
modprobed-db.service
user unit
can be queried.
Manually editing the database
Using the
#Automatic periodic database updates
or manually running
modprobed-db store
is not entirely foolproof:
Some modules get loaded then unloaded between two periodic database updates and may not be captured.
Some modules are used during boot and are not captured by
modprobed-db
(e.g.
efivarfs
), they need to be manually added to the
modprobed.db
file.
Thankfully, the
modprobed.db
database file is a simple text file that contains one kernel module name per line: it can be manually edited to add/remove a module. It is then recommended to run
modprobed-db store
after a manual edit so the modules get automatically reordered.
Recommended modules
These modules are recommended to be added to the
$XDG_CONFIG_HOME/modprobed.db
database file as they are broadly used:
ext4
,
fat
,
vfat
: to support common
file systems
and the
EFI system partition
nls_ascii
,
nls_cp437
: to support mounting the
EFI system partition
. This is required on
linux-xanmod
AUR
, on other kernels they may be built-in.
loop
: to mount a file as a file system
isofs
: to mount ISO files
cifs
: for network filesystems like
Samba
efivarfs
: for mounting the
UEFI#UEFI variables
file system
usb_storage
: to support
USB storage devices
usbhid
: to support USB input devices
For a ready-made database of commonplace modules,
linux-tkg
maintains one for each (recent) kernel version:
[1]
Building a kernel with modprobed-db
After the database has been adequately populated, it can be read directly by
make localmodconfig
.
Traditional compilation
modprobed-db
naturally intervenes in a
traditional compilation workflow
during the configuration step with the
default Arch .config file
.
Using the official Arch kernel PKGBUILD
The official Arch kernel
PKGBUILD
can be modified as shown to do this automatically:
...
    msg2 "Applying patch $src..."
    patch -Np1 < "../$src"
  done

  msg2 "Setting config..."
  cp ../config .config
  make olddefconfig
yes "" | make LSMOD=$HOME/.config/modprobed.db localmodconfig
make -s kernelrelease > ../version
...
Using linux-tkg
linux-tkg
offers a user-friendly kernel build script that also includes extra patches oriented towards improving desktop/gaming performance. In its configuration file, it supports user-provided
modprobed-db
's database files, but also offers its own "diet" database files: contains the most common modules while still producing a lighter kernel.
Benefits of modprobed-db with "make localmodconfig" in custom kernels
Significantly reduced compilation time
Unneeded modules are not built, the saved disk space is negligible for desktop computers as it is a relatively small gain (few hundred megabytes).
Comparisons using kernel version 5.13.1, where a
Kernel/Traditional compilation
is made with the default Arch configuration.
Machine CPU
# of threads
Compiler
make localmodconfig
# of Modules
Total Compilation Time
Kernel Compilation Time
Modules Compilation Time
Ryzen 5950X @ 4.55 GHz
32
GCC 11.1.0
No
5442
5m 12s
58s
4m 14s
Ryzen 5950X @ 4.55 GHz
32
GCC 11.1.0
Yes
227
1m 32s
57s
35s
Ryzen 5950X @ 4.55 GHz
32
Clang 12.0.1
No
5442
9m 5s
1m 13s
7m 52s
Ryzen 5950X @ 4.55 GHz
32
Clang 12.0.1
Yes
227
2m 13s
1m 13s
1m
Note
These results do not prove that GCC is faster than Clang at building the kernel. The GCC version used in these benchmarks is self built and optimized.
The main results of the benchmark is that 80% of the build time of a "full" kernel is spent on modules. Given that only a fraction of those modules are needed by any given machine, the build time can be reduced by ~70%. The results will vary from one machine to another but should be similar.
The number of modules can be determined by the following:
$ cd /lib/modules/15.13.1-your-custom-kernel
$ find -name '*.ko*' | wc -l
The kernel build time is obtained with:
$ time make -jx bzImage # Replace "x" with the wanted number of threads
then the modules build time is obtained with:
$ time make -jx modules # Replace "x" with the wanted number of threads
See also
https://docs.kernel.org/admin-guide/README.html#configuring-the-kernel
for documentation for
make localmodconfig
https://docs.kernel.org/kbuild/kconfig.html
and the parent directory