internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: systemd/mkosi

Issue:
gpg: signing failed: Inappropriate ioctl for device
When `mkosi` is called via SSH on a remote host, the following error occurs, while signing the archive or image:
`gpg: signing failed: Inappropriate ioctl for device`

This error can be fixed with the following line in the remot shell:
`export GPG_TTY=$(tty)`

It would be nice if `mkosi` would set this line to mitigate the issue. (I know the bug is clearly on the side of GPG.. sigh..)


Discussion:
I guess we can do this. Would be even better if we can detect remote shell and only apply in that case. PRs appreciated!
@DaanDeMeyer We can check for `SSH_CLIENT` `SSH_TTY` or `SSH_CONNECTION`. If we encounter one of those environment variables we are in an SSH session.

So everything what it would need is max 10 lines of code that check for one of these variables and then setting `GPG_TTY=$(tty)`

Resolving patch:
diff --git a/mkosi/__init__.py b/mkosi/__init__.py
--- a/mkosi/__init__.py
+++ b/mkosi/__init__.py
@@ -859,20 +859,18 @@ def calculate_signature(state: MkosiState) -> None:
             state.staging / state.config.output_checksum,
         ]
 
+        # Set the path of the keyring to use based on the environment if possible and fallback to the default
+        # path. Without this the keyring for the root user will instead be used which will fail for a
+        # non-root build.
+        env = dict(GNUPGHOME=os.environ.get("GNUPGHOME", os.fspath(((Path(os.environ["HOME"]) / ".gnupg")))))
+        if sys.stderr.isatty():
+            env |= dict(GPGTTY=os.ttyname(sys.stderr.fileno()))
+
         run(
             cmdline,
             # Do not output warnings about keyring permissions
             stderr=subprocess.DEVNULL,
-            env={
-                # Set the path of the keyring to use based on the environment
-                # if possible and fallback to the default path. Without this the
-                # keyring for the root user will instead be used which will fail
-                # for a non-root build.
-                "GNUPGHOME": os.environ.get(
-                    "GNUPGHOME",
-                    os.fspath(((Path(os.environ["HOME"]) / ".gnupg"))),
-                )
-            },
+            env=env,
         )
 
 


Test patch:
Repository: systemd/mkosi

Issue:
Build fails if host does not have /etc/gshadow
From my research, `/etc/gshadow` seems to be entirely optional. Both openSUSE and NixOS do not create `/etc/gshadow`.

This code expects gshadow to exist on the host and bind mounts `/dev/null` -> `/etc/gshadow` in bubblewrap if the path does not exist in the sysroot we are building:

https://github.com/systemd/mkosi/blob/9ae416df714cc87efad876b076ca10b677ed7036/mkosi/run.py#L338-L353

If `/etc/gshadow` does not exist on the host, bubblewrap first needs to create it before bind mounting it. This fails, since we do not have permissions to write to `/etc`.

A possible fix is to add `--tmpfs /etc` to the bwrap arguments and add all paths from the host as bind mounts (except for `/etc/passwd`,  `/etc/group`, `/etc/shadow`, `/etc/gshadow`). This way, we have permissions to create the mount for `/dev/null` -> `/etc/gshadow`.
Alternatively, we can maybe skip the bind mount if the file doesn't exist on the host?

Would you be interested in a PR that implements either the tmpfs mount for `/etc` or one that skips the mount if the file doesn't exist on the host?


Discussion:
Let's just skip the mount if it doesn't exist on the host. gshadow isn't terribly important and I'm pretty sure we'll still get what we want even without it mounted.

Resolving patch:
diff --git a/mkosi/run.py b/mkosi/run.py
--- a/mkosi/run.py
+++ b/mkosi/run.py
@@ -344,6 +344,8 @@ def finalize_passwd_mounts(root: Path) -> list[PathString]:
     options: list[PathString] = []
 
     for f in ("passwd", "group", "shadow", "gshadow"):
+        if not (Path("etc") / f).exists():
+            continue
         p = root / "etc" / f
         if p.exists():
             options += ["--bind", p, f"/etc/{f}"]


Test patch:
Repository: systemd/mkosi

Issue:
Images include dbus machine-id
In v15 it was changed such that images no longer ship a `/etc/machine-id`. While this is great in theory, the images still contain a dbus machine-id which is used by systemd to populate the normal machine-id if set.
Adding `RemoveFiles=/var/lib/dbus/machine-id` to the config fixed this and machines get proper random ids on bootup. It would be great if this file could also be excluded from the final images by default.


Discussion:

Resolving patch:
diff --git a/mkosi/__init__.py b/mkosi/__init__.py
--- a/mkosi/__init__.py
+++ b/mkosi/__init__.py
@@ -157,7 +157,10 @@ def install_distribution(state: MkosiState) -> None:
             if state.config.packages:
                 state.config.distribution.install_packages(state, state.config.packages)
 
-    for f in ("var/lib/systemd/random-seed", "var/lib/systemd/credential.secret", "etc/machine-info"):
+    for f in ("var/lib/systemd/random-seed",
+              "var/lib/systemd/credential.secret",
+              "etc/machine-info",
+              "var/lib/dbus/machine-id"):
         # Using missing_ok=True still causes an OSError if the mount is read-only even if the
         # file doesn't exist so do an explicit exists() check first.
         if (state.root / f).exists():


Test patch:
Repository: systemd/mkosi

Issue:
ManifestFormat should default to empty
By default `ManifestFormat` is set to `json`; when using `--format=directory` from the command line, this produces an extra file, which the user probably doesn't want if they're just trying to make a directory tree. One can pass `--manifest-format=''` to avoid that but it's not the easiest to discover, so I'd recommend defaulting this to the empty string instead so that no manifest is generated unless requested.


Discussion:
Also even with `--manifest-format=''` the log says `‣  Recording packages in manifest…`; this step should probably be skipped entirely if no manifest is being produced.

Resolving patch:
diff --git a/mkosi/__init__.py b/mkosi/__init__.py
--- a/mkosi/__init__.py
+++ b/mkosi/__init__.py
@@ -1270,7 +1270,10 @@ def dir_size(path: Union[Path, os.DirEntry[str]]) -> int:
     return dir_sum
 
 
-def save_manifest(state: MkosiState, manifest: Manifest) -> None:
+def save_manifest(state: MkosiState, manifest: Optional[Manifest]) -> None:
+    if not manifest:
+        return
+
     if manifest.has_data():
         if ManifestFormat.json in state.config.manifest_format:
             with complete_step(f"Saving manifest {state.config.output_manifest}"):
@@ -1749,7 +1752,7 @@ def finalize_staging(state: MkosiState) -> None:
 
 
 def build_image(args: MkosiArgs, config: MkosiConfig) -> None:
-    manifest = Manifest(config)
+    manifest = Manifest(config) if config.manifest_format else None
     workspace = tempfile.TemporaryDirectory(dir=config.workspace_dir, prefix=".mkosi-tmp")
 
     with workspace, scopedenv({"TMPDIR" : workspace.name}):
diff --git a/mkosi/config.py b/mkosi/config.py
--- a/mkosi/config.py
+++ b/mkosi/config.py
@@ -924,7 +924,6 @@ class MkosiConfigParser:
             metavar="FORMAT",
             section="Output",
             parse=config_make_list_parser(delimiter=",", parse=make_enum_parser(ManifestFormat)),
-            default=[ManifestFormat.json],
             help="Manifest Format",
         ),
         MkosiConfigSetting(


Test patch:
Repository: systemd/mkosi

Issue:
tput smam breaks build
Using latest on Debian Sid.

```
‣  Running finalize script…
‣  Creating tar archive /home/ander/Desktop/mkosi/tools/mkosi.workspace/.mkosi-tmp9zitpbja/staging/image.tar…
‣  /home/ander/Desktop/mkosi/tools/mkosi.output/image size is 1016.1M, consumes 1016.1M.
‣ "tput smam" returned non-zero exit code 1.
make: *** [Makefile:13: build] Error 1
```

`tput` in Debian do not understand `smam`?

```
$ tput smam; echo $?
1
```

Introduced in e651b88.

Ugly workaround:
```
$ ln -s /usr/bin/true ~/.local/bin/tput
```


Discussion:

Resolving patch:
diff --git a/mkosi/__main__.py b/mkosi/__main__.py
--- a/mkosi/__main__.py
+++ b/mkosi/__main__.py
@@ -51,8 +51,8 @@ def main() -> None:
         run_verb(args, presets)
     finally:
         if sys.stderr.isatty() and shutil.which("tput"):
-            run(["tput", "cnorm"])
-            run(["tput", "smam"])
+            run(["tput", "cnorm"], check=False)
+            run(["tput", "smam"], check=False)
 
 
 if __name__ == "__main__":


Test patch:
Repository: systemd/mkosi

Issue:
`mkosi ssh` does not work on Fedora host
When invoking the ssh command on a Fedora 38 host I get the following error message (with `--debug`):

```
‣ + ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o LogLevel=ERROR -o ProxyCommand=socat - VSOCK-CONNECT:3436578145:%p root@mkosi
Bad owner or permissions on /etc/ssh/ssh_config.d/50-redhat.conf
```

Running the shown ssh command from the terminal itself works without problems.

---

PS: It would also be great if the debug log would quote command parts if they contain spaces. In this case that would be: `ssh ... -o "ProxyCommand=socat - VSOCK-CONNECT:3436578145:%p" ...`


Discussion:

Resolving patch:
diff --git a/mkosi/qemu.py b/mkosi/qemu.py
--- a/mkosi/qemu.py
+++ b/mkosi/qemu.py
@@ -336,6 +336,7 @@ def run_qemu(args: MkosiArgs, config: MkosiConfig) -> None:
 def run_ssh(args: MkosiArgs, config: MkosiConfig) -> None:
     cmd = [
         "ssh",
+        "-F", "none",
         # Silence known hosts file errors/warnings.
         "-o", "UserKnownHostsFile=/dev/null",
         "-o", "StrictHostKeyChecking=no",


Test patch:
Repository: systemd/mkosi

Issue:
[Meta] declare a policy about adding new distributions
Before people start creating issues asking to support their favorite distribution, I think that mkosi should declare its policy regarding new distributions support.

The policy should state in which terms (if any) you will be willing to support a new distributions.


Discussion:
yeah, it's a difficult question where to draw the line. And I figure when adding this we should also open the door for smaller distros to hook things up to mkosi too, by adding some form of drop-in logic, where additional distros could be added by dropping in some minimal file that does the minimal core bits to hook things up...

Resolving patch:
diff --git a/mkosi/distributions/__init__.py b/mkosi/distributions/__init__.py
--- a/mkosi/distributions/__init__.py
+++ b/mkosi/distributions/__init__.py
@@ -72,6 +72,8 @@ def tools_tree_packages(cls) -> list[str]:
 
 
 class Distribution(StrEnum):
+    # Please consult docs/distribution-policy.md and contact one
+    # of the mkosi maintainers before implementing a new distribution.
     fedora       = enum.auto()
     debian       = enum.auto()
     ubuntu       = enum.auto()


Test patch:
Repository: systemd/mkosi

Issue:
--output behaves inconsistently with directories
Running something like `mkosi --distribution fedora --release 38 --architecture x86-64 --format directory --output x64` works just fine, but using `--output /tmp/x64` (or any other fully qualified path) instead fails with `‣ /tmp/x64 is not a valid filename`.


Discussion:
--output only configures the filename. --output-directory configures the output directory. Maybe we should rename --output to --output-name or so?
`--output-directory foo` works, but that's the _enclosing_ directory, so if you only pass that and don't pass output your tree ends up under `foo/image`. There doesn't seem to be a straightforward way to just say "put a tree under /foo/bar" which is what I want here.
That would be done using `--output-directory foo --output bar`. The reasoning behind this is that you can have multiple outputs so it felt more natural to split the output prefix (`--output`) and the output directory (`--output-directory`) into separate options. The other thing is that you generally end up reusing the output directory with different output prefixes when doing presets. But maybe we should look into translating output when given as a path into a separate directory and prefix automatically.
> But maybe we should look into translating output when given as a path into a separate directory and prefix automatically.

Yeah I think that would be nice. I understand the current behavior now, but for a user that's not familiar with it and just quickly wants a directory tree it's kinda confusing.

Resolving patch:
diff --git a/mkosi/config.py b/mkosi/config.py
--- a/mkosi/config.py
+++ b/mkosi/config.py
@@ -468,18 +468,14 @@ def config_parse_path(value: Optional[str], old: Optional[Path]) -> Optional[Pat
     return config_parse_path
 
 
-def config_parse_filename(value: Optional[str], old: Optional[str]) -> Optional[str]:
+def config_parse_output(value: Optional[str], old: Optional[str]) -> Optional[str]:
     if not value:
         return None
 
-    if value == "." or value == "..":
-        die(". and .. are not valid filenames")
-
-    if "/" in value:
-        die(
-            f"{value!r} is not a valid filename. "
-            "(Output= requires a filename with no path components, relative to output directory.)"
-        )
+    if value == "." or value == ".." or "/" in value:
+        die(f"{value!r} is not a valid filename.",
+            hint="Output= or --output= requires a filename with no path components. "
+                 "Use OutputDirectory= or --output-dir= to configure the output directory.")
 
     return value
 
@@ -1147,7 +1143,7 @@ def parse_ini(path: Path, only_sections: Collection[str] = ()) -> Iterator[tuple
         metavar="NAME",
         section="Output",
         specifier="o",
-        parse=config_parse_filename,
+        parse=config_parse_output,
         help="Output name",
     ),
     MkosiConfigSetting(


Test patch:
Repository: systemd/mkosi

Issue:
Add useradd script.
This supports --root so we should probably add a script to add --root automatically.


Discussion:

Resolving patch:
diff --git a/mkosi/__init__.py b/mkosi/__init__.py
--- a/mkosi/__init__.py
+++ b/mkosi/__init__.py
@@ -364,8 +364,12 @@ def finalize_host_scripts(
     state: MkosiState,
     helpers: dict[str, Sequence[PathString]],  # FIXME: change dict to Mapping when PyRight is fixed
 ) -> contextlib.AbstractContextManager[Path]:
-    git = {"git": ("git", "-c", "safe.directory=*")} if find_binary("git") else {}
-    return finalize_scripts(git | helpers | package_manager_scripts(state))
+    scripts: dict[str, Sequence[PathString]] = {}
+    if find_binary("git"):
+        scripts["git"] = ("git", "-c", "safe.directory=*")
+    if find_binary("useradd"):
+        scripts["useradd"] = ("useradd", "--root", state.root)
+    return finalize_scripts(scripts | helpers | package_manager_scripts(state))
 
 
 def finalize_chroot_scripts(state: MkosiState) -> contextlib.AbstractContextManager[Path]:


Test patch:
Repository: systemd/mkosi

Issue:
Unknown setting genkey-valid-days
Hi, I want to generate secure boot key with `mkosi --genkey-valid-days=3560 genkey`. But I got
```shell
‣ Unknown setting --genkey-valid-days
```

Previously using mkosi version 14, I could use `mkosi --secure-boot-valid-days=3650 genkey`. I see in the [mkosi.md](https://github.com/systemd/mkosi/blob/main/mkosi/resources/mkosi.md), it was changed to `--genkey-valid-days`.

Thanks.


Discussion:

Resolving patch:
diff --git a/mkosi/config.py b/mkosi/config.py
--- a/mkosi/config.py
+++ b/mkosi/config.py
@@ -2135,14 +2135,12 @@ def create_argument_parser(action: type[argparse.Action]) -> argparse.ArgumentPa
         "--genkey-valid-days",
         metavar="DAYS",
         help="Number of days keys should be valid when generating keys",
-        action=action,
         default="730",
     )
     parser.add_argument(
         "--genkey-common-name",
         metavar="CN",
         help="Template for the CN when generating keys",
-        action=action,
         default="mkosi of %u",
     )
     parser.add_argument(


Test patch:
Repository: systemd/mkosi

Issue:
Debian tools which use /etc/alternatives break
Regression caused by https://github.com/systemd/mkosi/pull/2201, already reported in https://github.com/systemd/mkosi/issues/2206.

When putting a keyring in `/etc/apt/trusted.gpg.d`, eventually `apt-key` will be called, which will make a call to `awk`. On debian that is installed as
```
lrwxrwxrwx 1 65534 65534 21 Jun 17  2022 /usr/bin/awk -> /etc/alternatives/awk
```
Since `/etc/alternatives` is not mounted, the call to awk fails and thus apt thinks there was no valid key found.


Discussion:

Resolving patch:
diff --git a/mkosi/bubblewrap.py b/mkosi/bubblewrap.py
--- a/mkosi/bubblewrap.py
+++ b/mkosi/bubblewrap.py
@@ -86,6 +86,7 @@ def bwrap(
         # This mount is writable so bwrap can create extra directories or symlinks inside of it as needed. This isn't a
         # problem as the package manager directory is created by mkosi and thrown away when the build finishes.
         "--bind", state.pkgmngr / "etc", "/etc",
+        "--ro-bind-try", "/etc/alternatives", "/etc/alternatives",
         "--bind", "/var/tmp", "/var/tmp",
         "--bind", "/tmp", "/tmp",
         "--bind", Path.cwd(), Path.cwd(),


Test patch:
Repository: systemd/mkosi

Issue:
Show failed command also before entering debug shell
In the debug shell one often retries the failed command with different flags or higher verbosity. It would thus be very useful to show the failed command also before, or instead of, only after leaving the debug shell.


Discussion:

Resolving patch:
diff --git a/mkosi/run.py b/mkosi/run.py
--- a/mkosi/run.py
+++ b/mkosi/run.py
@@ -313,6 +313,8 @@ def preexec() -> None:
     except FileNotFoundError as e:
         die(f"{e.filename} not found.")
     except subprocess.CalledProcessError as e:
+        if log:
+            log_process_failure(cmdline, e.returncode)
         if ARG_DEBUG_SHELL.get():
             subprocess.run(
                 [*sandbox, "sh"],
@@ -325,8 +327,6 @@ def preexec() -> None:
                 cwd=cwd,
                 preexec_fn=preexec,
             )
-        if log:
-            log_process_failure(cmdline, e.returncode)
         # Remove the sandboxing stuff from the command line to show a more readable error to users.
         e.cmd = cmdline
         raise


Test patch:
Repository: systemd/mkosi

Issue:
`mkosi qemu` crashes when `--directory` has only one path component
Starting with version 20 (`20.1-1~bpo12+1` from Debian bookworm-backports), `mkosi qemu` seems to assume that its config directory path always has at least 2 components. This leads to crashes like the following:

```
root@mkosi:~# mkdir /foo; mkosi --directory=/foo qemu
[...]
mke2fs 1.47.0 (5-Feb-2023)
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/mkosi/run.py", line 162, in uncaught_exception_handler
    yield
  File "/usr/lib/python3.11/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/mkosi/__main__.py", line 34, in main
    run_verb(args, images)
  File "/usr/lib/python3/dist-packages/mkosi/__init__.py", line 3468, in run_verb
    {
  File "/usr/lib/python3/dist-packages/mkosi/qemu.py", line 830, in run_qemu
    sandbox=config.sandbox(network=True, devices=True, relaxed=True),
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/mkosi/config.py", line 1386, in sandbox
    return sandbox_cmd(
           ^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/mkosi/sandbox.py", line 124, in sandbox_cmd
    if (d := os.fspath(list(Path.cwd().parents)[-2])) not in (*dirs, "/home", "/usr", "/nix", "/tmp"):
                       ~~~~~~~~~~~~~~~~~~~~~~~~^^^^
IndexError: list index out of range
```

I ran into this problem by using mkosi inside a Vagrant VM, which mounts the project directory to `/vagrant` by default. Changing that mountpoint to `/media/vagrant` works as a workaround.




Discussion:

Resolving patch:
diff --git a/mkosi/sandbox.py b/mkosi/sandbox.py
--- a/mkosi/sandbox.py
+++ b/mkosi/sandbox.py
@@ -119,9 +119,16 @@ def sandbox_cmd(
             if Path(d).exists():
                 cmdline += ["--bind", d, d]
 
-        # `Path.parents` only supports slices and negative indexing from Python 3.10 onwards.
-        # TODO: Remove list() when we depend on Python 3.10 or newer.
-        if (d := os.fspath(list(Path.cwd().parents)[-2])) not in (*dirs, "/home", "/usr", "/nix", "/tmp"):
+        if len(Path.cwd().parents) >= 2:
+            # `Path.parents` only supports slices and negative indexing from Python 3.10 onwards.
+            # TODO: Remove list() when we depend on Python 3.10 or newer.
+            d = os.fspath(list(Path.cwd().parents)[-2])
+        elif len(Path.cwd().parents) == 1:
+            d = os.fspath(Path.cwd())
+        else:
+            d = ""
+
+        if d and d not in (*dirs, "/home", "/usr", "/nix", "/tmp"):
             cmdline += ["--bind", d, d]
 
     if vartmp:


Test patch:
Repository: systemd/mkosi

Issue:
`sh` unavailable in sandbox on non `usr`-merged systems
mkosi's sandbox [mounts `/usr:/usr`](https://github.com/systemd/mkosi/blob/14b07c74d5b3a8b0f44d9a5c733f7902c7859417/mkosi/sandbox.py#L88) if no ToolsTree is specified. On NixOS, which is not usr-merged, this causes problems when programs have runtime dependencies on absolute paths in `(/usr)/bin`. This is the case for `apt-get install`, that tries to find an  `sh` at runtime in `/usr/bin` or `/bin`. That fails on NixOS, due to only `/usr` being mounted, while `sh` is in `/bin/sh`.

Two possible workarounds that I have in mind:
1. (Preferred) Find a way to tell `apt-get` which shell to use. `dpkg`  [respects the `SHELL` environment variable](https://man7.org/linux/man-pages/man1/dpkg.1.html), but I couldn't find a similar option for `apt-get`.  If there would be such an option, we should simply tell `apt-get` to use the `sh` from the Nix store that's already mounted.
2. Loosen the sandbox and mount `/bin(/sh)` into the environment. 

If you have any input regarding the first proposed solution, let me know. Otherwise, I'd be happy to implement the second solution too.

Cheers, Moritz


Discussion:

Resolving patch:
diff --git a/mkosi/sandbox.py b/mkosi/sandbox.py
--- a/mkosi/sandbox.py
+++ b/mkosi/sandbox.py
@@ -137,6 +137,8 @@ def sandbox_cmd(
     for d in ("bin", "sbin", "lib", "lib32", "lib64"):
         if (p := tools / d).is_symlink():
             cmdline += ["--symlink", p.readlink(), Path("/") / p.relative_to(tools)]
+        elif p.is_dir():
+            cmdline += ["--ro-bind", p, Path("/") / p.relative_to(tools)]
 
     path = "/usr/bin:/usr/sbin" if tools != Path("/") else os.environ["PATH"]
 


Test patch: