Repository: OSInside/kiwi

Issue:
Missing package metadata in built Ubuntu-based image
Hi,

I’m building a disk image based on Ubuntu 24.04 LTS using KIWI NG version 10.2.16. The generated image boots and appears functional, but the package management system (APT/dpkg) is in an inconsistent state.

Specifically, many binaries are present on the system, but dpkg does not list their corresponding packages as installed. For example:
/usr/bin/bash exists, but dpkg -l does not show the bash package
/usr/bin/find exists, but dpkg -l does not list findutils
There are many other similar cases.

It seems that these binaries are part of packages that KIWI installs during the bootstrap phase, but since they are not declared in the `<packages type="image">` section, they do not appear in the resulting dpkg database of the image.

Please let me know if you'd like me to share an example config.xml to help reproduce the issue.

Thanks!

Hints from issue comments:
@schaefi: @koprok yes your observations are correct. I believe this depends on the way we bootstrap. See the code here

* https://github.com/OSInside/kiwi/blob/main/kiwi/package_manager/apt.py#L207

So for bootstrap we invoke apt such that it resolves a list of given bootstrap packages and only downloads them. The downloaded .deb's are then extracted via `dpkg-deb --fsys-tarfile` and we run the package scripts. This procedure does not create a proper "apt database" and is basically the result of apt missing a "--root" option.

A nasty workaround is to add the bootstrap packages also to the type="image" section such that they get properly installed by apt again even though they were extracted during bootstrap.

So I believe all you have said is correct but I don't know a nice solution to the problem.

Ideas welcome@koprok: Hi @schaefi,

I think I found a solution similar to the "nasty workaround" you suggested. I noticed in the KIWI log that the bootstrap phase installs a specially-named package, `?essential`. By adding this package to my `<packages type="image">` section, like so:
```
<packages type="image">
    <package name="?essential"/> <!-- Fixes missing bootstrap packages metadata -->
    <!-- ... other packages ... -->
</packages>
```
The issue appears to be resolved.

I understand that this approach causes the packages installed during bootstrap to be reinstalled during the image phase. Do you see any issues with this?
The good thing is that I don't need to specify the individual packages, which may change between Ubuntu releases. Adding `?essential` ensures that the same set of bootstrap packages are installed without having to list them all manually.
Maybe KIWI could implicitly add `?essential`, what do you think?
@schaefi: @koprok actually a nice idea. I coded a PR that adds all bootstrap packages (including the fixed ?essential) to the image phase such that they get reinstalled properly

>Do you see any issues with this?

To be honest; I don't know :) Actually I don't think it will cause any issue. But we have to verify it. Therefore I will zip a kiwi Staging build including the open PR and that triggers all of our runtime builds. If all is well there we can wait for the community review and I hope someone with deeper knowledge on Debian distros can have a look too.

Let's see what they say@schaefi: So far all Debian based integration tests are passing

Solution patch:
diff --git a/kiwi/package_manager/apt.py b/kiwi/package_manager/apt.py
index 12f01f20..cce860dd 100644
--- a/kiwi/package_manager/apt.py
+++ b/kiwi/package_manager/apt.py
@@ -72,6 +72,7 @@ class PackageManagerApt(PackageManagerBase):
         self.command_env = runtime_config['command_env']
         self.distribution = runtime_config['distribution']
         self.distribution_path = runtime_config['distribution_path']
+        self.bootstrap_packages: List[str] = []
 
     def request_package(self, name: str) -> None:
         """
@@ -243,14 +244,16 @@ class PackageManagerApt(PackageManagerBase):
                 'Apt update: {0} {1}'.format(result.output, result.error)
             )
             package_names = Temporary(prefix='kiwi_debs_').new_file()
+            self.bootstrap_packages = [
+                '?essential'
+            ] + self.package_requests
             download_bootstrap = [
                 'apt-get'
             ] + self.apt_get_args + self.custom_args + [
                 'install',
                 '-oDebug::pkgDPkgPm=1',
-                f'-oDPkg::Pre-Install-Pkgs::=cat >{package_names.name}',
-                '?essential'
-            ] + self.package_requests
+                f'-oDPkg::Pre-Install-Pkgs::=cat >{package_names.name}'
+            ] + self.bootstrap_packages
             # Download solved bootstrap packages
             result = Command.run(
                 download_bootstrap, self.command_env
@@ -319,6 +322,13 @@ class PackageManagerApt(PackageManagerBase):
         )
         apt_get_command.extend(self.custom_args)
         apt_get_command.append('install')
+        # Due to the special bootstrap process, the packages unpacked
+        # during bootstrap are not properly listed in the apt index.
+        # Therefore the bootstrap packages are added to the install
+        # phase which causes an install of this packages again to
+        # fix the apt index and provide a consistent system from
+        # an apt perspective.
+        apt_get_command.extend(self.bootstrap_packages)
         apt_get_command.extend(self._package_requests())
 
         return Command.call(


Test patch:
diff --git a/test/unit/package_manager/apt_test.py b/test/unit/package_manager/apt_test.py
index 28fd8286..5af23204 100644
--- a/test/unit/package_manager/apt_test.py
+++ b/test/unit/package_manager/apt_test.py
@@ -258,11 +258,14 @@ class TestPackageManagerApt:
     @patch('kiwi.command.Command.run')
     def test_process_install_requests(self, mock_run, mock_call):
         self.manager.request_package('vim')
+        self.manager.bootstrap_packages = ['?essential', 'some']
         self.manager.process_install_requests()
-        mock_call.assert_called_once_with([
-            'chroot', 'root-dir', 'apt-get',
-            '-c', 'apt.conf', '-y', 'install', 'vim'],
-            self.env
+        mock_call.assert_called_once_with(
+            [
+                'chroot', 'root-dir', 'apt-get',
+                '-c', 'apt.conf', '-y', 'install',
+                '?essential', 'some', 'vim'
+            ], self.env
         )
 
     @patch('kiwi.command.Command.call')