Repository: systemd/mkosi

Issue:
Empty rpmdb after building older RPM distro on Fedora 36 host
I've tried latest git 92e021d on a Fedora 36 host and the rpm db changes are causing me grief building older rpm-based distros.

While the install is fine, if you have to add additional DNF repos to install further packages in the prepare script, the system doesn't know what is installed as the host's rpm has put the the sqlite rpmdb in /usr/lib/sysimage/rpm/ which the container's rpm doesn't parse or look for.

I've seen d3d0fbd4 and 0497f2bf before it which rebuild the rpm db inside the container but as they don't know what's actually installed, this results in an empty database which is fairly useless for further installs in the prepare script.

I've not found a specific bug for this, but wondering if I'm missing something here. If this can be fixed and released in Fedora 36, it would certainly help.

Current workaround is is to only use ```Packages=dnf``` in the mkosi.default then do ```dnf --releasever=8 install dnf ...``` in the container. This will reinstall dnf and deps inside the container and mean the rpmdb is properly populated for the container distro. This is pretty rubbish workaround as no caching is used for packages which means everything needs to be downloaded again etc which is far from ideal.

Are there any other better workarounds/fixes?


Discussion:
This is semi-related to #805 in that we're at the whims of the host's RPM/DNF packages/environment. 

I'm not sure what the solution is other than making mkosi build some kind of staging container in which you install rpm/dnf from the target distro (but in your native arch if different from the specified arch) before linking up the cache dirs to that container and using it to actually install the packages for your new container.

Would seem like a fair amount of architectural rework (and I'm not sure it'll help with the user issue sadly)!
Just to make sure I understand the problem properly: This is an upstream RPM problem and doesn't work with any mkosi version on Fedora 36 properly and the recent touch ups around rpm on our side are insufficient?
> Just to make sure I understand the problem properly: This is an upstream RPM problem and doesn't work with any mkosi version on Fedora 36 properly and the recent touch ups around rpm on our side are insufficient?

Certainly as far as I can tell, this is a correct statement.

Installing a very basic container using mkosi from git on a Fedora 36 host and the following mkosi.default:
````
[Distribution]
Distribution=rocky_epel
Release=8
Architecture=x86_64

[Packages]
Packages=rpm,dnf

[Output]
Format=directory
Output=ostree
````

When entering that container via ```systemd-nspawn -D ostree``` running ```rpm -qa``` gives no results i.e. the rpm database is blank.

The rpm version on rocky/centos 8 does link with sqlite as a backend but not sure it will look in /usr/lib/sysimage/rpm/ for the files, so the ```rpm --rebuilddb``` that is issued in d3d0fbd and 0497f2b will only serve to initialise a blank rpmdb AFAICT.

This also means dnf can't seem to know what release it's running under either:

```
[root@ostree ~]# dnf install dnf
Unable to detect release version (use '--releasever' to specify release version)
Rocky Linux $releasever - AppStream             2.3 kB/s | 5.7 kB     00:02    
Errors during downloading metadata for repository 'appstream':
  - Status code: 404 for https://mirrors.rockylinux.org/mirrorlist?arch=x86_64&repo=AppStream-$releasever (IP: 199.232.54.132)
Error: Failed to download metadata for repo 'appstream': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.rockylinux.org/mirrorlist?arch=x86_64&repo=AppStream-$releasever (IP: 199.232.54.132)
```

Using ```--releasever 8``` does fix this up and we can initialise things properly.
I think I've found a work around (for rocky/centos etc. 8 at least):

```
[root@ostree ~]# rm -rf /var/lib/rpm/*
[root@ostree ~]# cp /usr/lib/sysimage/rpm/* /var/lib/rpm/
[root@ostree ~]# rpm --rebuilddb
warning: Found SQLITE rpmdb.sqlite database while attempting bdb backend: using sqlite backend.
[root@ostree ~]# rpm -qa
gpg-pubkey-2f86d6a1-5cf7cefb
...
```

So I will work on a patch for mkosi to try and detect this format and run those fix up commands prior to the rebuild db.

Hmm, there is a fundamental chicken+egg issue here I think.

As I noted in #934 I don't think it's right to migrate the rpmdb in invoke_dnf() as this will break remove_packages() invocation (and manifest.record_packages()) which is run later using the host's rpm.

So we could clean this all up after the manifest is written, but then this means we cannot have a nice migrated rpm db when we are running our prepare script to install additional repos/modules etc.

Of course if we could do more to specify repos/modules as part of the initial install packages phase, this would negate the need to do any package management during prepare, but it's still probably necessary in some edge cases.

So I don't really know what's best here. Something is always going to break!

I think for now with my use case, I'll do the file move + --rebuilddb myself in the prepare script, knowing that I do not write a manifest nor define any packages to remove and I should be OK for my uses cases. Sadly this isn't really an upstream fix, nor do I know what such a fix would look like (other than my suggestion to build a minimal initial+native arch container to install the package manager and then run it from there rather than the host machine - but I'm sure that will come with it's own version compatibility headaches.


For completeness, this is the snippet I put in my (final) mkosi.prepare:

```
if [ -f /usr/bin/rpm -a -d /usr/lib/sysimage/rpm ]; then
  echo "Migrating Fedora 36+ RPM DB to correct location"
  rm -f /var/lib/rpm/*
  cp /usr/lib/sysimage/rpm/rpmdb* /var/lib/rpm/
  rpm --rebuilddb --define "_db_backend bdb" || exit 1
  rm -rf /usr/lib/sysimage/rpm
fi
```
The "proper" solution here is likely to copy mock which first installs dnf/rpm in an intermediate bootstrap chroot and use those to install all the other packages (credits to @keszybz for the suggestion). So we'd use host dnf to install only dnf in a separate chroot, and then use `run_workspace_command()` to run dnf in the chroot and bind mount the final chroot directory, cache dirs and other stuff in there.
> The "proper" solution here is likely to copy mock which first installs dnf/rpm in an intermediate bootstrap chroot and use those to install all the other packages (credits to @keszybz for the suggestion). So we'd use host dnf to install only dnf in a separate chroot, and then use `run_workspace_command()` to run dnf in the chroot and bind mount the final chroot directory, cache dirs and other stuff in there.

Yeah that's what I was thinking (what I tried to describe above). Probably a bit too much work for me to look at just now (certainly in the short term, I'm afraid)

Resolving patch:
diff --git a/mkosi/__init__.py b/mkosi/__init__.py
--- a/mkosi/__init__.py
+++ b/mkosi/__init__.py
@@ -1943,111 +1943,97 @@ def make_rpm_list(args: MkosiArgs, packages: Set[str], do_run_build_script: bool
     return packages
 
 
-def clean_dnf_metadata(root: Path, always: bool) -> None:
-    """Remove dnf metadata if /bin/dnf is not present in the image
+def flatten(lists: Iterable[Iterable[T]]) -> List[T]:
+    """Flatten a sequence of sequences into a single list."""
+    return list(itertools.chain.from_iterable(lists))
 
-    If dnf is not installed, there doesn't seem to be much use in
-    keeping the dnf metadata, since it's not usable from within the
-    image anyway.
-    """
-    paths = [
-        root / "var/lib/dnf",
-        *root.glob("var/log/dnf.*"),
-        *root.glob("var/log/hawkey.*"),
-        root / "var/cache/dnf",
-    ]
 
-    cond = always or not os.access(root / "bin/dnf", os.F_OK, follow_symlinks=False)
+def clean_paths(
+        root: Path,
+        globs: Sequence[str],
+        tool: str,
+        always: bool) -> None:
+    """Remove globs under root if always or if tool is not found under root."""
 
-    if not cond or not any(path.exists() for path in paths):
+    toolp = root / tool.lstrip('/')
+    cond = always or not os.access(toolp, os.F_OK, follow_symlinks=False)
+
+    paths = flatten(root.glob(glob.lstrip('/')) for glob in globs)
+
+    if not cond or not paths:
         return
 
-    with complete_step("Cleaning dnf metadata…"):
+    with complete_step(f"Cleaning {toolp.name} metadata…"):
         for path in paths:
             unlink_try_hard(path)
 
 
-def clean_yum_metadata(root: Path, always: bool) -> None:
-    """Remove yum metadata if /bin/yum is not present in the image"""
+def clean_dnf_metadata(root: Path, always: bool) -> None:
+    """Remove dnf metadata if /bin/dnf is not present in the image
+
+    If dnf is not installed, there doesn't seem to be much use in keeping the
+    dnf metadata, since it's not usable from within the image anyway.
+    """
     paths = [
-        root / "var/lib/yum",
-        *root.glob("var/log/yum.*"),
-        root / "var/cache/yum",
+        "/var/lib/dnf",
+        "/var/log/dnf.*",
+        "/var/log/hawkey.*",
+        "/var/cache/dnf",
     ]
 
-    cond = always or not os.access(root / "bin/yum", os.F_OK, follow_symlinks=False)
+    clean_paths(root, paths, tool='/bin/dnf', always=always)
 
-    if not cond or not any(path.exists() for path in paths):
-        return
 
-    with complete_step("Cleaning yum metadata…"):
-        for path in paths:
-            unlink_try_hard(path)
+def clean_yum_metadata(root: Path, always: bool) -> None:
+    """Remove yum metadata if /bin/yum is not present in the image"""
+    paths = [
+        "/var/lib/yum",
+        "/var/log/yum.*",
+        "/var/cache/yum",
+    ]
+
+    clean_paths(root, paths, tool='/bin/yum', always=always)
 
 
 def clean_rpm_metadata(root: Path, always: bool) -> None:
     """Remove rpm metadata if /bin/rpm is not present in the image"""
-    path = root / "var/lib/rpm"
-
-    cond = always or not os.access(root / "bin/rpm", os.F_OK, follow_symlinks=False)
-
-    if not cond or not path.exists():
-        return
+    paths = [
+        "/var/lib/rpm",
+        "/usr/lib/sysimage/rpm",
+    ]
 
-    with complete_step("Cleaning rpm metadata…"):
-        unlink_try_hard(path)
+    clean_paths(root, paths, tool='/bin/rpm', always=always)
 
 
 def clean_tdnf_metadata(root: Path, always: bool) -> None:
-    """Remove tdnf metadata if /bin/tdnf is not present in the image"""
+    """Remove tdnf metadata if /usr/bin/tdnf is not present in the image"""
     paths = [
-        *root.glob("var/log/tdnf.*"),
-        root / "var/cache/tdnf",
+        "/var/log/tdnf.*",
+        "/var/cache/tdnf",
     ]
 
-    cond = always or not os.access(root / "usr/bin/tdnf", os.F_OK, follow_symlinks=False)
-
-    if not cond or not any(path.exists() for path in paths):
-        return
-
-    with complete_step("Cleaning tdnf metadata…"):
-        for path in paths:
-            unlink_try_hard(path)
+    clean_paths(root, paths, tool='/usr/bin/tdnf', always=always)
 
 
 def clean_apt_metadata(root: Path, always: bool) -> None:
     """Remove apt metadata if /usr/bin/apt is not present in the image"""
     paths = [
-        root / "var/lib/apt",
-        root / "var/log/apt",
-        root / "var/cache/apt",
+        "/var/lib/apt",
+        "/var/log/apt",
+        "/var/cache/apt",
     ]
 
-    cond = always or not os.access(root / "usr/bin/apt", os.F_OK, follow_symlinks=False)
-
-    if not cond or not any(path.exists() for path in paths):
-        return
-
-    with complete_step("Cleaning apt metadata…"):
-        for path in paths:
-            unlink_try_hard(path)
+    clean_paths(root, paths, tool='/usr/bin/apt', always=always)
 
 
 def clean_dpkg_metadata(root: Path, always: bool) -> None:
     """Remove dpkg metadata if /usr/bin/dpkg is not present in the image"""
     paths = [
-        root / "var/lib/dpkg",
-        root / "var/log/dpkg.log",
+        "/var/lib/dpkg",
+        "/var/log/dpkg.log",
     ]
 
-    cond = always or not os.access(root / "usr/bin/dpkg", os.F_OK, follow_symlinks=False)
-
-    if not cond or not any(path.exists() for path in paths):
-        return
-
-    with complete_step("Cleaning dpkg metadata…"):
-        for path in paths:
-            unlink_try_hard(path)
+    clean_paths(root, paths, tool='/usr/bin/dpkg', always=always)
 
 
 def clean_package_manager_metadata(args: MkosiArgs, root: Path) -> None:
@@ -2063,12 +2049,13 @@ def clean_package_manager_metadata(args: MkosiArgs, root: Path) -> None:
 
     # we try then all: metadata will only be touched if any of them are in the
     # final image
-    clean_dnf_metadata(root, always=args.clean_package_metadata is True)
-    clean_yum_metadata(root, always=args.clean_package_metadata is True)
-    clean_rpm_metadata(root, always=args.clean_package_metadata is True)
-    clean_tdnf_metadata(root, always=args.clean_package_metadata is True)
-    clean_apt_metadata(root, always=args.clean_package_metadata is True)
-    clean_dpkg_metadata(root, always=args.clean_package_metadata is True)
+    always = args.clean_package_metadata is True
+    clean_dnf_metadata(root, always=always)
+    clean_yum_metadata(root, always=always)
+    clean_rpm_metadata(root, always=always)
+    clean_tdnf_metadata(root, always=always)
+    clean_apt_metadata(root, always=always)
+    clean_dpkg_metadata(root, always=always)
     # FIXME: implement cleanup for other package managers: swupd, pacman
 
 
@@ -2142,6 +2129,17 @@ def invoke_dnf(
         shutil.move(cast(str, rpmdb_home), rpmdb)
 
 
+def link_rpm_db(root: Path) -> None:
+    """Link /var/lib/rpm to /usr/lib/sysimage/rpm for compat with old rpm"""
+    rpmdb = root / "usr/lib/sysimage/rpm"
+    rpmdb_old = root / "var/lib/rpm"
+    if rpmdb.exists() and not rpmdb_old.is_symlink():
+        # We create the symlink in exactly the same fashion that Fedora do
+        with complete_step("Creating compat symlink /var/lib/rpm → /usr/lib/sysimage/rpm"):
+            unlink_try_hard(rpmdb_old)
+            rpmdb_old.symlink_to("../../usr/lib/sysimage/rpm")
+
+
 def install_packages_dnf(
     args: MkosiArgs,
     root: Path,
@@ -3290,6 +3288,11 @@ def install_distribution(args: MkosiArgs, root: Path, do_run_build_script: bool,
     with mount_cache(args, root):
         install[args.distribution](args, root, do_run_build_script)
 
+    # Link /var/lib/rpm→/usr/lib/sysimage/rpm for compat with old rpm.
+    # We do this only if the new location is used, which depends on the dnf
+    # version and configuration on the host. Thus we do this reactively, after the
+    # installation has completed.
+    link_rpm_db(root)
 
 def remove_packages(args: MkosiArgs, root: Path) -> None:
     """Remove packages listed in args.remove_packages"""
diff --git a/mkosi/manifest.py b/mkosi/manifest.py
--- a/mkosi/manifest.py
+++ b/mkosi/manifest.py
@@ -88,7 +88,6 @@ def record_rpm_packages(self, root: Path) -> None:
             ["rpm", f"--root={root}", "-qa", "--qf",
              r"%{NEVRA}\t%{SOURCERPM}\t%{NAME}\t%{ARCH}\t%{SIZE}\t%{INSTALLTIME}\n"],
             stdout=PIPE,
-            stderr=DEVNULL,
             text=True,
         )
 
@@ -140,7 +139,6 @@ def record_deb_packages(self, root: Path) -> None:
             ["dpkg-query", f"--admindir={root}/var/lib/dpkg", "--show", "--showformat",
              r'${Package}\t${source:Package}\t${Version}\t${Architecture}\t${Installed-Size}\t${db-fsys:Last-Modified}\n'],
             stdout=PIPE,
-            stderr=DEVNULL,
             text=True,
         )
 


Test patch: