Repository: systemd/mkosi

Issue:
Support libarchive cpio
Hi,

please support libarchive cpio(bsdcpio), which does not have `--reproducible`

https://lists.gnu.org/archive/html/bug-cpio/2014-11/msg00000.html

https://github.com/systemd/mkosi/blob/2c45d0effb1871750a2e9f897510d2745cb6d6b9/mkosi/__init__.py#L3489


Discussion:
I'd review a PR for this, but I'm not entirely sold, that this is a good idea. Why isn't using the GNU cpio an option or why can't BSD cpio implement these options? Making this configurable would make the output unreproducible and I don't think this is were everybody's headed nowadays. Granted, unreproducible output would be preferable to no output at all, if GNU cpio really is not an option, but some background would be helpful.
@behrmann First there are some benefits of having libarchive as replacement for GNU archiving tools.
Well my attached thread for GNU ML explains couple of things. Second by default libarchive cpio does provide reproducible archives
gah, which distros install /usr/bin/cpio as the bsd version? This is a mess to support. what's the point? different for the sake of doing things differently?

do these distros make the gnu version available as gcpio or so? if so, I'd just change mkosi to search for "gcpio" first, and use "cpio" only as fallback. I'd rather have systematic behaviour  so that we use the same tools on all distros, then to add messy quirks for the bad choices of specific distros...
> @behrmann First there are some benefits of having libarchive as replacement for GNU archiving tools. Well my attached thread for GNU ML explains couple of things. Second by default libarchive cpio does provide reproducible archives

It would have helped to get a link to the message that actually said something about libarchive's version, instead of me having to read through the whole thread only to find the info the the very last message :)

> gah, which distros install /usr/bin/cpio as the bsd version? This is a mess to support. what's the point? different for the sake of doing things differently?
> 
> do these distros make the gnu version available as gcpio or so? if so, I'd just change mkosi to search for "gcpio" first, and use "cpio" only as fallback. I'd rather have systematic behaviour so that we use the same tools on all distros, then to add messy quirks for the bad choices of specific distros...

I'd assume Mandriva does this, since this already prompted the default to `gtar` wit `tar_binary()`. I guess doing the same for `cpio` would be the consistent solution.
> gah, which distros install /usr/bin/cpio as the bsd version? This is a mess to support. what's the point? different for the sake of doing things differently?
> 
Yes OpenMandriva does that, same as we have LLVM/clang as default $CC/$CXX along with GNU/GCC (for stubborn software)  or GNU/glibc with musl :) The freedom of choice.

> do these distros make the gnu version available as gcpio or so?
yes package is called gnucpio and binary is gcpio
https://github.com/OpenMandrivaAssociation/gnucpio/blob/16c4deaf3588eabb4d5a2b2f563e89caa94fe3a7/gnucpio.spec#L60




> > @behrmann First there are some benefits of having libarchive as replacement for GNU archiving tools. Well my attached thread for GNU ML explains couple of things. Second by default libarchive cpio does provide reproducible archives
> 
> It would have helped to get a link to the message that actually said something about libarchive's version, instead of me having to read through the whole thread only to find the info the the very last message :)
> 
Sorry for that, i should have mention that all the details are on the end of this thread.
According to this doc https://github.com/libarchive/libarchive/wiki/FormatCpio, a libarchive's cpio uses PAX which is reproducible by default.
> 
> I'd assume Mandriva does this, since this already prompted the default to `gtar` wit `tar_binary()`. I guess doing the same for `cpio` would be the consistent solution.
Indeed yes.


> Yes OpenMandriva does that, same as we have LLVM/clang as default $CC/$CXX along with GNU/GCC (for stubborn software) or GNU/glibc with musl :) The freedom of choice.

the choice of pain maybe...

Anyway, I think we should just do what we do for tar, and prefer gcpio over cpio, and avoid the mess.
(not sure why libarchive even bothers with a cpio drop-in if it is not compatible, but well, I am sure mandriva has a reason to create synthetic problems like that)
> the choice of pain maybe..

No pain no gain

> not sure why libarchive even bothers with a cpio drop-in if it is not compatible

Well libarchive is POSIX compliant so i guess that is why.
In general, if we support libarchive, we should support it directly. No need to shell out to a helper if we can avoid it.

Nevertheless, in this specific case, I think that I tried to use libarchive initially, and the output was not to the kernels liking, see the description for 5b97f8f2f4309062cdf29b31cca2d3a5153be0c2. I think it'd be nice to see this, but somebody would need to figure out what differs between the current output and libarchive output.

Resolving patch:
diff --git a/mkosi/archive.py b/mkosi/archive.py
--- a/mkosi/archive.py
+++ b/mkosi/archive.py
@@ -1,13 +1,28 @@
 # SPDX-License-Identifier: LGPL-2.1+
 
 import os
+import shutil
 from collections.abc import Iterable
 from pathlib import Path
 from typing import Optional
 
 from mkosi.log import log_step
 from mkosi.run import bwrap, finalize_passwd_mounts
-from mkosi.util import tar_binary
+
+
+def tar_binary() -> str:
+    # Some distros (Mandriva) install BSD tar as "tar", hence prefer
+    # "gtar" if it exists, which should be GNU tar wherever it exists.
+    # We are interested in exposing same behaviour everywhere hence
+    # it's preferable to use the same implementation of tar
+    # everywhere. In particular given the limited/different SELinux
+    # support in BSD tar and the different command line syntax
+    # compared to GNU tar.
+    return "gtar" if shutil.which("gtar") else "tar"
+
+
+def cpio_binary() -> str:
+    return "gcpio" if shutil.which("gcpio") else "cpio"
 
 
 def tar_exclude_apivfs_tmp() -> list[str]:
@@ -74,7 +89,7 @@ def make_cpio(src: Path, dst: Path, files: Optional[Iterable[Path]] = None) -> N
     log_step(f"Creating cpio archive {dst}…")
     bwrap(
         [
-            "cpio",
+            cpio_binary(),
             "--create",
             "--reproducible",
             "--null",
diff --git a/mkosi/util.py b/mkosi/util.py
--- a/mkosi/util.py
+++ b/mkosi/util.py
@@ -14,7 +14,6 @@
 import pwd
 import re
 import resource
-import shutil
 import stat
 import sys
 import tempfile
@@ -223,17 +222,6 @@ def values(cls) -> list[str]:
         return list(map(str, cls))
 
 
-def tar_binary() -> str:
-    # Some distros (Mandriva) install BSD tar as "tar", hence prefer
-    # "gtar" if it exists, which should be GNU tar wherever it exists.
-    # We are interested in exposing same behaviour everywhere hence
-    # it's preferable to use the same implementation of tar
-    # everywhere. In particular given the limited/different SELinux
-    # support in BSD tar and the different command line syntax
-    # compared to GNU tar.
-    return "gtar" if shutil.which("gtar") else "tar"
-
-
 def one_zero(b: bool) -> str:
     return "1" if b else "0"
 


Test patch: