Repository: systemd/mkosi

Issue:
Auto-Installed information is lost in apt-based and dnf-based distro images
### mkosi commit the issue has been seen with

main

### Used host distribution

Any

### Used target distribution

Debian

### Linux kernel version used

Any

### CPU architectures issue was seen on

Any

### Unexpected behaviour you saw

After building a Debian image, running `apt-mark showauto` inside it shows empty list. Dependencies installed during the build process are not listed as automatic.

Running `apt remove <Package> --autoremove` inside the generated image does not function correctly. It fails to identify and remove orphaned dependencies that should be cleaned up.

The root cause seems to be the `/var/lib/apt/extended_states` file is not preserved during the build process, adding this `mkosi.postinst` script will fix this issue (but I think it might be better to include it during the build process):

```
#!/bin/bash

cp /var/lib/apt/extended_states $BUILDROOT/var/lib/apt/
```



### Used mkosi config

```ini
[Distribution]
Distribution=debian
Release=testing

[Content]
CleanPackageMetadata=no
Packages=systemd
    systemd-sysv
    systemd-resolved
    systemd-cron
    login
    util-linux
    coreutils
    bash
    apt
    # Used to test apt remove --autoremove inside the image.
    fish

[Output]
Format=disk
```

### `apt remove fish` output

```sh
REMOVING:                       
  fish*

Summary:
  Upgrading: 0, Installing: 0, Removing: 1, Not Upgrading: 0
  Freed space: 8784 kB

Continue? [Y/n]
```

### Expected `apt remove fish` output
```
The following packages were automatically installed and are no longer required:
  fish-common  libncursesw6   libpython3-stdlib      libsqlite3-0  netbase          python3.13
  groff-base   libpcre2-32-0  libpython3.13-minimal  libuchardet0  procps           python3.13-minimal
  libffi8      libpipeline1   libpython3.13-stdlib   man-db        python3          readline-common
  libgdbm6t64  libproc2-0     libreadline8t64        media-types   python3-minimal
Use 'apt autoremove' to remove them.

REMOVING:
  fish*

Summary:
  Upgrading: 0, Installing: 0, Removing: 1, Not Upgrading: 0
  Freed space: 8784 kB

Continue? [Y/n]
```

Hints from issue comments:
@MoltenArmor: I've done some similar tests on Fedora:

`mkosi.conf`:
```
[Build]
ToolsTree=default

[Distribution]
Distribution=fedora
Release=41

[Content]
CleanPackageMetadata=no
Packages=systemd
    util-linux
    coreutils
    bash
    dnf5
   # Used to test dnf remove nano.
    nano

[Output]
Format=disk
```

And similar problem (`dnf autoremove` does not work) happens on Fedora, too. This is because `/var/lib/dnf/history.sqlite*` files are not preserved in the image.

`dnf repoquery --userinstalled` output:

```
alternatives-0:1.31-1.fc41.aarch64
audit-libs-0:4.0.3-1.fc41.aarch64
authselect-0:1.5.0-8.fc41.aarch64
authselect-libs-0:1.5.0-8.fc41.aarch64
basesystem-0:11-21.fc41.noarch
bash-0:5.2.32-1.fc41.aarch64
bzip2-libs-0:1.0.8-19.fc41.aarch64
ca-certificates-0:2024.2.69_v8.0.401-1.0.fc41.noarch
coreutils-0:9.5-11.fc41.aarch64
coreutils-common-0:9.5-11.fc41.aarch64
cracklib-0:2.9.11-6.fc41.aarch64
crypto-policies-0:20250124-1.git4d262e7.fc41.noarch
curl-0:8.9.1-3.fc41.aarch64
cyrus-sasl-lib-0:2.1.28-27.fc41.aarch64
dbus-1:1.14.10-4.fc41.aarch64
......    # All packages installed
```

`dnf remove nano` output:

```
Package                                  Arch        Version                                  Repository                 Size
Removing:
 nano                                    aarch64     8.1-1.fc41                               <unknown>               3.0 MiB

Transaction Summary:
 Removing:           1 package

After this operation, 3 MiB will be freed (install 0 B, remove 3 MiB).
Is this ok [y/N]
```

Expected `dnf remove nano` output, with running `cp -r /var/lib/dnf $BUILDROOT/var/lib/` inside `mkosi.postinst`:

```
Package                                  Arch        Version                                  Repository                 Size
Removing:
 nano                                    aarch64     8.1-1.fc41                               fedora                  3.0 MiB
Removing unused dependencies:
 file-libs                               aarch64     5.45-7.fc41                              fedora                 10.0 MiB

Transaction Summary:
 Removing:           2 packages

After this operation, 13 MiB will be freed (install 0 B, remove 13 MiB).
Is this ok [y/N]
```

This issue does not appear when building with dnf5, because it uses `/usr/lib/sysimage/libdnf5/transaction_history.sqlite` to keep history, which is preserved.

**If this is the intended behavior, please feel free to close this issue**.@septatrix: In general if you mix mkosi and apt/dnf inside the image you are likely having a hard time, especially once kernel updates etc come into the picture. I think the workaround you use (`cp /var/lib/apt/extended_states $BUILDROOT/var/lib/apt/ `) likely just copies the file from you host to the image so is also probably not what you want.@s3rj1k: @MoltenArmor I can confirm bug on Debian and proposed fix, building on ArchLinux host Debian Image, so `/var/lib/apt/extended_states` is definitely not coming from HostOS@MoltenArmor: I've also done some tests on Archlinux images. However, this problem does not occur.

`mkosi.conf`:

```
[Build]
ToolsTree=default

[Distribution]
Distribution=arch

[Content]
CleanPackageMetadata=no
Packages=systemd
    util-linux
    coreutils
    bash
    pacman
    # Used to test pacman -R vim
    vim

[Output]
Format=directory
```

`pacman -Qdt` output after running `pacman -R vim`:

```
gpm 1.20.7.r38.ge82d1a6-6
vim-runtime 9.1.1236-1
```

And `pacman -Qe` output is also "correct" (from my POV):

```
bash 5.2.037-2
coreutils 9.6-4
filesystem 2024.11.21-1
pacman 7.0.0.r6.gc685ae6-2
systemd 257.5-1
util-linux 2.41-4
```

This is probably because pacman uses files in `/var/lib/pacman/local/*` to determine which packages are "explicitly installed", and these files are preserved in the image. This results in incosistency between Archlinux images and other distro images.@MoltenArmor: Similarly, the same issue does not occur in OpenSUSE images:

`mkosi.conf`:
```
[Build]
ToolsTree=default

[Distribution]
Distribution=opensuse

[Content]
CleanPackageMetadata=no
Packages=systemd
    util-linux
    coreutils
    bash
    zypper
    # Used to test zypper remove vim
    vim

[Output]
Format=directory
```

`zypper packages --userinstalled` works correctly (I think):

```
S  | Repository | Name       | Version     | Arch
---+------------+------------+-------------+--------
i+ | OSS        | bash       | 5.2.37-18.1 | aarch64
i+ | OSS        | coreutils  | 9.6-1.4     | aarch64
i+ | OSS        | filesystem | 84.87-18.3  | aarch64
i+ | OSS        | glibc      | 2.41-2.1    | aarch64
i+ | OSS        | systemd    | 257.4-1.2   | aarch64
i+ | OSS        | util-linux | 2.40.4-4.4  | aarch64
i+ | OSS        | zypper     | 1.14.88-1.1 | aarch64
```

And after removing `vim`, `zypper packages --unneeded` also works correctly (I think):

```
S  | Repository | Name            | Version      | Arch
---+------------+-----------------+--------------+--------
i  | OSS        | perl            | 5.40.1-2.1   | aarch64
i  | OSS        | vim-data-common | 9.1.1258-1.1 | noarch
i  | OSS        | xxd             | 9.1.1258-1.1 | aarch64
```

According to [this link](https://www.reddit.com/r/openSUSE/comments/1dz1627/where_is_zyppers_manually_installed_information/), it seems this is because `/var/lib/zypp/AutoInstalled` and `/var/log/zypp/history` are both (or at least the latter) preserved in the image.@MoltenArmor: For `apt` we can probably just add `-o Dir::State::extended_states=/buildroot/var/lib/apt/extended_states` to `apt-get`.

However, it seems 
`dnf5`'s `transaction_history_dir` option is not available for `dnf`, which makes the situation more complicated.@kolonelkrazy: > For `apt` we can probably just add `-o Dir::State::extended_states=/buildroot/var/lib/apt/extended_states` to `apt-get`.
> 
> However, it seems dnf5's `transaction_history_dir` option is not available for dnf, which makes the situation more complicated.

This is a more proper way compared to your original solution. 

In general this should  be an option somewhere, there are cases where you would and wouldn't want this history.


One shot replication for the original report for anyone interested:
```bas
cd $(mktemp -d) && { tee mkosi.conf  <<-'EOF'
[Distribution]
Distribution=ubuntu
Release=noble
Repositories=restricted universe multiverse

[Output]
OutputDirectory=mkosi.output
Format=directory

[Content]
CleanPackageMetadata=no
Packages=systemd
    systemd-sysv
    systemd-resolved
    systemd-cron
    login
    util-linux
    coreutils
    bash
    apt
    # Used to test apt remove --autoremove inside the image.
    fish
Autologin=yes
EOF
} && mkosi build && file mkosi.output/image/var/lib/apt/extended_state
```@s3rj1k: > wouldn't want this history

pretty sure you would want to know what packages are dependencies of selected packages, in this case from mkosi config, in all possible cases

Solution patch:
diff --git a/mkosi/__init__.py b/mkosi/__init__.py
index b7cbbbe1..7b649fd5 100644
--- a/mkosi/__init__.py
+++ b/mkosi/__init__.py
@@ -69,10 +69,10 @@ from mkosi.config import (
     Verity,
     Vmm,
     cat_config,
+    dump_json,
     expand_delayed_specifiers,
+    finalize_configdir,
     format_bytes,
-    get_configdir,
-    have_history,
     in_sandbox,
     parse_boolean,
     parse_config,
@@ -578,7 +578,7 @@ def finalize_host_scripts(
 @contextlib.contextmanager
 def finalize_config_json(config: Config) -> Iterator[Path]:
     with tempfile.NamedTemporaryFile(mode="w") as f:
-        f.write(config.to_json())
+        f.write(dump_json(config.to_dict()))
         f.flush()
         yield Path(f.name)
 
@@ -620,7 +620,7 @@ def run_configure_scripts(config: Config) -> Config:
                             *sources,
                         ],
                     ),
-                    input=config.to_json(indent=None),
+                    input=dump_json(config.to_dict(), indent=None),
                     stdout=subprocess.PIPE,
                 )  # fmt: skip
 
@@ -3873,30 +3873,22 @@ def copy_repository_metadata(config: Config, dst: Path) -> None:
     subdir = config.distribution.package_manager(config).subdir(config)
 
     with complete_step("Copying repository metadata"):
-        for d in ("cache", "lib"):
-            src = config.package_cache_dir_or_default() / d / subdir
-            if not src.exists():
-                logging.debug(f"{src} does not exist, not copying repository metadata from it")
-                continue
+        cachedir = config.package_cache_dir_or_default() / "cache" / subdir
+        if cachedir.exists():
+            with umask(~0o755):
+                (dst / "cache" / subdir).mkdir(parents=True, exist_ok=True)
 
             with tempfile.TemporaryDirectory() as tmp:
                 os.chmod(tmp, 0o755)
 
                 # cp doesn't support excluding directories but we can imitate it by bind mounting
                 # an empty directory over the directories we want to exclude.
-                exclude: list[PathString]
-                if d == "cache":
-                    exclude = flatten(
-                        ("--ro-bind", tmp, workdir(p))
-                        for p in config.distribution.package_manager(config).cache_subdirs(src)
-                    )
-                else:
-                    exclude = flatten(
-                        ("--ro-bind", tmp, workdir(p))
-                        for p in config.distribution.package_manager(config).state_subdirs(src)
-                    )
+                exclude = flatten(
+                    ("--ro-bind", tmp, workdir(p))
+                    for p in config.distribution.package_manager(config).package_subdirs(cachedir)
+                )
 
-                subdst = dst / d / subdir
+                subdst = dst / "cache" / subdir
                 with umask(~0o755):
                     subdst.mkdir(parents=True, exist_ok=True)
 
@@ -3906,7 +3898,25 @@ def copy_repository_metadata(config: Config, dst: Path) -> None:
                 ) -> AbstractContextManager[list[PathString]]:
                     return config.sandbox(options=[*options, *exclude])
 
-                copy_tree(src, subdst, sandbox=sandbox)
+                copy_tree(cachedir, subdst, sandbox=sandbox)
+        else:
+            logging.debug(f"{cachedir} does not exist, not copying repository metadata from it")
+
+        statedir = config.package_cache_dir_or_default() / "lib" / subdir
+
+        with umask(~0o755):
+            (dst / "lib" / subdir).mkdir(parents=True, exist_ok=True)
+
+        for src in config.distribution.package_manager(config).state_subdirs(statedir):
+            if not src.exists():
+                logging.debug(f"{src} does not exist, not copying repository metadata from it")
+                continue
+
+            subdst = dst / "lib" / subdir / src.relative_to(statedir)
+            with umask(~0o755):
+                subdst.mkdir(parents=True, exist_ok=True)
+
+            copy_tree(src, subdst, sandbox=config.sandbox)
 
 
 @contextlib.contextmanager
@@ -3979,10 +3989,7 @@ def build_image(context: Context) -> None:
         check_root_populated(context)
         run_build_scripts(context)
 
-        if context.config.output_format == OutputFormat.none or (
-            context.args.rerun_build_scripts
-            and (context.config.output_dir_or_cwd() / context.config.output).exists()
-        ):
+        if context.config.output_format == OutputFormat.none or context.args.rerun_build_scripts:
             return
 
         if wantrepo:
@@ -4489,7 +4496,7 @@ def generate_key_cert_pair(args: Args) -> None:
     keylength = 2048
     expiration_date = datetime.date.today() + datetime.timedelta(int(args.genkey_valid_days))
 
-    configdir = get_configdir(args)
+    configdir = finalize_configdir(args)
 
     for f in (configdir / "mkosi.key", configdir / "mkosi.crt"):
         if f.exists() and not args.force:
@@ -4525,29 +4532,11 @@ def generate_key_cert_pair(args: Args) -> None:
     )  # fmt: skip
 
 
-def bump_image_version() -> None:
-    """Write current image version plus one to mkosi.version"""
-
-    version_file = Path("mkosi.version")
-    if not version_file.exists():
-        die(f"Cannot bump image version, '{version_file}' not found")
-
-    if os.access(version_file, os.X_OK):
-        die(f"Cannot bump image version, '{version_file}' is executable")
-
-    version = version_file.read_text().strip()
-    v = version.split(".")
-
-    try:
-        v[-1] = str(int(v[-1]) + 1)
-    except ValueError:
-        v += ["2"]
-        logging.warning("Last component of current version is not a decimal integer, appending '.2'")
-
-    new_version = ".".join(v)
-
-    logging.info(f"Bumping version: '{version}' → '{new_version}'")
-    version_file.write_text(f"{new_version}\n")
+def finalize_image_version(args: Args, config: Config) -> None:
+    p = finalize_configdir(args) / "mkosi.version"
+    assert config.image_version
+    p.write_text(config.image_version)
+    logging.info(f"Wrote new version {config.image_version} to {p}")
 
 
 def check_workspace_directory(config: Config) -> None:
@@ -4665,8 +4654,11 @@ def validate_certificates_and_keys(config: Config) -> None:
 
 
 def needs_build(args: Args, config: Config, force: int = 1) -> bool:
+    if args.rerun_build_scripts:
+        return False
+
     return (
-        (args.force >= force and not args.rerun_build_scripts)
+        (args.force >= force)
         or not (config.output_dir_or_cwd() / config.output_with_compression).exists()
         # When the output is a directory, its name is the same as the symlink we create that points to the
         # actual output when not building a directory. So if the full output path exists, we have to check
@@ -4857,7 +4849,7 @@ def sync_repository_metadata(
                 )
 
     src = last.package_cache_dir_or_default() / "cache" / subdir
-    for p in last.distribution.package_manager(last).cache_subdirs(src):
+    for p in last.distribution.package_manager(last).package_subdirs(src):
         p.mkdir(parents=True, exist_ok=True)
 
     # If we're in incremental mode and caching metadata is not explicitly disabled, cache the keyring and the
@@ -4939,10 +4931,6 @@ def run_build(
         )
 
 
-def dump_json(dict: dict[str, Any]) -> str:
-    return json.dumps(dict, cls=JsonEncoder, indent=4, sort_keys=True)
-
-
 def run_verb(args: Args, tools: Optional[Config], images: Sequence[Config], *, resources: Path) -> None:
     images = list(images)
 
@@ -4973,9 +4961,6 @@ def run_verb(args: Args, tools: Optional[Config], images: Sequence[Config], *, r
     if args.verb == Verb.genkey:
         return generate_key_cert_pair(args)
 
-    if args.verb == Verb.bump:
-        return bump_image_version()
-
     if args.verb == Verb.dependencies:
         _, _, [deps] = parse_config(
             ["--directory=", "--repositories=", *args.cmdline, "--include=mkosi-tools", "build"],
@@ -5016,6 +5001,10 @@ def run_verb(args: Args, tools: Optional[Config], images: Sequence[Config], *, r
     # The images array has been modified so we need to reevaluate last again.
     last = images[-1]
 
+    if args.verb == Verb.bump:
+        finalize_image_version(args, last)
+        return
+
     if args.verb == Verb.clean:
         if tools and args.force > 0:
             run_clean(args, tools)
@@ -5077,11 +5066,6 @@ def run_verb(args: Args, tools: Optional[Config], images: Sequence[Config], *, r
 
         _, _, manifest = cache_tree_paths(tools)
         manifest.write_text(dump_json(tools.cache_manifest()))
-        Path(".mkosi-private/history/tools.json").unlink(missing_ok=True)
-
-    if tools and last.history and not Path(".mkosi-private/history/tools.json").exists():
-        Path(".mkosi-private/history").mkdir(parents=True, exist_ok=True)
-        Path(".mkosi-private/history/tools.json").write_text(dump_json(tools.to_dict()))
 
     if args.verb.needs_tools():
         return {
@@ -5108,12 +5092,7 @@ def run_verb(args: Args, tools: Optional[Config], images: Sequence[Config], *, r
         logging.info(f"Output path {output} exists already. (Use --force to rebuild.)")
         return
 
-    if (
-        args.rerun_build_scripts
-        and not args.force
-        and last.output_format != OutputFormat.none
-        and not output.exists()
-    ):
+    if args.rerun_build_scripts and last.output_format != OutputFormat.none and not output.exists():
         die(
             f"Image '{last.image}' must be built once before --rerun-build-scripts can be used",
             hint="Build the image once with 'mkosi build'",
@@ -5146,9 +5125,7 @@ def run_verb(args: Args, tools: Optional[Config], images: Sequence[Config], *, r
                     hint="Add the &I specifier to the cache key to avoid this issue",
                 )
 
-    if last.is_incremental() and (
-        last.incremental == Incremental.strict or (args.rerun_build_scripts and not args.force)
-    ):
+    if last.is_incremental() and (last.incremental == Incremental.strict or args.rerun_build_scripts):
         if args.force > 1:
             die(
                 "Cannot remove incremental caches when building with Incremental=strict",
@@ -5179,27 +5156,22 @@ def run_verb(args: Args, tools: Optional[Config], images: Sequence[Config], *, r
             if args.force > 1 or not have_cache(initrd):
                 remove_cache_entries(initrd)
 
-    if not have_history(args):
-        for i, config in enumerate(images):
-            if args.verb != Verb.build:
-                check_tools(config, args.verb)
+    for i, config in enumerate(images):
+        if args.verb != Verb.build:
+            check_tools(config, args.verb)
 
-            images[i] = config = run_configure_scripts(config)
+        images[i] = config = run_configure_scripts(config)
 
-        # The images array has been modified so we need to reevaluate last again.
-        # Also ensure that all other images are reordered in case their dependencies were modified.
-        last = images[-1]
-        images = resolve_deps(images[:-1], last.dependencies) + [last]
+    # The images array has been modified so we need to reevaluate last again.
+    # Also ensure that all other images are reordered in case their dependencies were modified.
+    last = images[-1]
+    images = resolve_deps(images[:-1], last.dependencies) + [last]
 
     if (
         args.rerun_build_scripts
         or last.output_format == OutputFormat.none
         or not (last.output_dir_or_cwd() / last.output).exists()
     ):
-        history = (last.output_format == OutputFormat.none and not args.rerun_build_scripts) or (
-            last.output_format != OutputFormat.none and not (last.output_dir_or_cwd() / last.output).exists()
-        )
-
         for config in images:
             if any(
                 source.type != KeySourceType.file
@@ -5280,13 +5252,7 @@ def run_verb(args: Args, tools: Optional[Config], images: Sequence[Config], *, r
                 logging.info("All images have already been built and do not have any build scripts")
 
         if args.auto_bump:
-            bump_image_version()
-
-        if last.history and history:
-            Path(".mkosi-private/history").mkdir(parents=True, exist_ok=True)
-            Path(".mkosi-private/history/latest.json").write_text(
-                dump_json({"Images": [config.to_dict() for config in images]})
-            )
+            finalize_image_version(args, last)
 
     if args.verb == Verb.build:
         return
diff --git a/mkosi/config.py b/mkosi/config.py
index f7fe14bb..cb1a520c 100644
--- a/mkosi/config.py
+++ b/mkosi/config.py
@@ -31,7 +31,7 @@ from pathlib import Path
 from typing import Any, Callable, ClassVar, Generic, Optional, Protocol, TypeVar, Union, cast
 
 from mkosi.distributions import Distribution, detect_distribution
-from mkosi.log import ARG_DEBUG, ARG_DEBUG_SANDBOX, ARG_DEBUG_SHELL, die
+from mkosi.log import ARG_DEBUG, ARG_DEBUG_SANDBOX, ARG_DEBUG_SHELL, complete_step, die
 from mkosi.pager import page
 from mkosi.run import SandboxProtocol, find_binary, nosandbox, run, sandbox_cmd, workdir
 from mkosi.sandbox import Style, __version__
@@ -1787,10 +1787,6 @@ class Args:
     def to_dict(self) -> dict[str, Any]:
         return dataclasses.asdict(self, dict_factory=dict_with_capitalised_keys_factory)
 
-    def to_json(self, *, indent: Optional[int] = 4, sort_keys: bool = True) -> str:
-        """Dump Args as JSON string."""
-        return json.dumps(self.to_dict(), cls=JsonEncoder, indent=indent, sort_keys=sort_keys)
-
     @classmethod
     def from_json(cls, s: Union[str, dict[str, Any], SupportsRead[str], SupportsRead[bytes]]) -> "Args":
         """Instantiate a Args object from a (partial) JSON dump."""
@@ -2347,12 +2343,15 @@ class Config:
 
         return d
 
-    def to_json(self, *, indent: Optional[int] = 4, sort_keys: bool = True) -> str:
-        """Dump Config as JSON string."""
-        return json.dumps(self.to_dict(), cls=JsonEncoder, indent=indent, sort_keys=sort_keys)
+    @classmethod
+    def to_partial_dict(cls, partial: dict[str, Any]) -> dict[str, Any]:
+        return dict_with_capitalised_keys_factory([(k, v) for k, v in partial.items() if k in cls.fields()])
 
     @classmethod
-    def from_json(cls, s: Union[str, dict[str, Any], SupportsRead[str], SupportsRead[bytes]]) -> "Config":
+    def from_partial_json(
+        cls,
+        s: Union[str, dict[str, Any], SupportsRead[str], SupportsRead[bytes]],
+    ) -> dict[str, Any]:
         """Instantiate a Config object from a (partial) JSON dump."""
         if isinstance(s, str):
             j = json.loads(s)
@@ -2382,9 +2381,13 @@ class Config:
                 )
 
         value_transformer = json_type_transformer(cls)
-        j = {(tk := key_transformer(k)): value_transformer(tk, v) for k, v in j.items()}
+        return {(tk := key_transformer(k)): value_transformer(tk, v) for k, v in j.items()}
 
-        return dataclasses.replace(cls.default(), **{k: v for k, v in j.items() if k in cls.fields()})
+    @classmethod
+    def from_json(cls, s: Union[str, dict[str, Any], SupportsRead[str], SupportsRead[bytes]]) -> "Config":
+        return dataclasses.replace(
+            cls.default(), **{k: v for k, v in cls.from_partial_json(s).items() if k in cls.fields()}
+        )
 
     def find_binary(self, *names: PathString, tools: bool = True) -> Optional[Path]:
         return find_binary(*names, root=self.tools() if tools else Path("/"), extra=self.extra_search_paths)
@@ -4518,7 +4521,6 @@ class ParseContext:
         self.defaults: dict[str, Any] = {}
         # Compare inodes instead of paths so we can't get tricked by bind mounts and such.
         self.includes: set[tuple[int, int]] = set()
-        self.only_sections: tuple[str, ...] = tuple()
 
     def setting_prohibited(self, setting: ConfigSetting[T]) -> bool:
         image = self.config["image"]
@@ -4799,9 +4801,6 @@ class ParseContext:
                 if self.setting_prohibited(s):
                     continue
 
-                if self.only_sections and s.section not in self.only_sections:
-                    continue
-
                 for f in s.path_suffixes:
                     f = f"mkosi.{f}"
 
@@ -4841,7 +4840,7 @@ class ParseContext:
 
             for section, k, v in parse_ini(
                 path,
-                only_sections=self.only_sections or {s.section for s in SETTINGS} | {"Host"},
+                only_sections={s.section for s in SETTINGS} | {"Host"},
             ):
                 if not k and not v:
                     continue
@@ -4907,13 +4906,30 @@ class ParseContext:
         return ns
 
 
-def have_history(args: Args, tools: bool = False) -> bool:
-    filename = "tools" if tools else "latest"
+def want_new_history(args: Args) -> bool:
+    if args.directory is None:
+        return False
+
+    if not args.verb.needs_build():
+        return False
+
+    if args.rerun_build_scripts:
+        return False
+
+    if args.verb != Verb.build and args.force == 0:
+        return False
+
+    return True
+
+
+def have_history(args: Args) -> bool:
+    if want_new_history(args):
+        return False
 
     if args.directory is None:
         return False
 
-    if args.verb in (Verb.build, Verb.cat_config, Verb.clean):
+    if args.verb in (Verb.clean, Verb.sandbox):
         return False
 
     if args.verb == Verb.summary and args.force > 0:
@@ -4922,10 +4938,13 @@ def have_history(args: Args, tools: bool = False) -> bool:
     if args.verb.needs_tools() and args.force > 0:
         return False
 
-    if args.verb.needs_build() and args.force > 0 and not args.rerun_build_scripts:
+    if args.verb.needs_build() and args.force > 0:
+        return False
+
+    if args.verb == Verb.build and not args.rerun_build_scripts:
         return False
 
-    return Path(f".mkosi-private/history/{filename}.json").exists()
+    return Path(".mkosi-private/history/latest.json").exists()
 
 
 def finalize_default_tools(
@@ -4979,7 +4998,7 @@ def finalize_default_tools(
     return Config.from_dict(context.finalize())
 
 
-def get_configdir(args: Args) -> Path:
+def finalize_configdir(args: Args) -> Path:
     """Allow locating all mkosi configuration in a mkosi/ subdirectory
     instead of in the top-level directory of a git repository.
     """
@@ -4993,11 +5012,40 @@ def get_configdir(args: Args) -> Path:
     return Path.cwd()
 
 
+def bump_image_version(configdir: Path) -> str:
+    version_file = configdir / "mkosi.version"
+    if os.access(version_file, os.X_OK):
+        die(f"Cannot bump image version, '{version_file}' is executable")
+
+    if version_file.exists():
+        version = version_file.read_text().strip()
+    else:
+        version = None
+
+    if (bump := configdir / "mkosi.bump").exists():
+        with complete_step(f"Running bump script {bump}"):
+            new_version = run([bump], stdout=subprocess.PIPE).stdout.strip()
+    elif version is not None:
+        v = version.split(".")
+
+        try:
+            v[-1] = str(int(v[-1]) + 1)
+        except ValueError:
+            v += ["2"]
+            logging.warning("Last component of current version is not a decimal integer, appending '.2'")
+
+        new_version = ".".join(v)
+    else:
+        new_version = "1"
+
+    logging.info(f"Bumping version: '{none_to_na(version)}' → '{new_version}'")
+    return new_version
+
+
 def parse_config(
     argv: Sequence[str] = (),
     *,
     resources: Path = Path("/"),
-    only_sections: Sequence[str] = (),
 ) -> tuple[Args, Optional[Config], tuple[Config, ...]]:
     argv = list(argv)
 
@@ -5025,6 +5073,9 @@ def parse_config(
     if args.rerun_build_scripts and not args.verb.needs_build():
         die(f"--rerun-build-scripts cannot be used with the '{args.verb}' command")
 
+    if args.rerun_build_scripts and args.force:
+        die("--force cannot be used together with --rerun-build-scripts")
+
     if args.cmdline and not args.verb.supports_cmdline():
         die(f"Arguments after verb are not supported for the '{args.verb}' command")
 
@@ -5040,14 +5091,7 @@ def parse_config(
         return args, None, ()
 
     if have_history(args):
-        try:
-            j = json.loads(Path(".mkosi-private/history/latest.json").read_text())
-            *subimages, prev = [Config.from_json(c) for c in j["Images"]]
-        except (KeyError, ValueError):
-            die(
-                "Unable to parse history from .mkosi-private/history/latest.json",
-                hint="Build with -f to generate a new history file from scratch",
-            )
+        history = Config.from_partial_json(Path(".mkosi-private/history/latest.json").read_text())
 
         # If we're operating on a previously built image (vm, boot, shell, ...), we're not rebuilding the
         # image and the configuration of the latest build is available, we load the config that was used to
@@ -5055,28 +5099,17 @@ def parse_config(
         # section settings which we allow changing without requiring a rebuild of the image.
         for s in SETTINGS:
             if s.section in ("Include", "Runtime"):
+                history.pop(s.dest, None)
                 continue
 
-            if s.dest in context.cli and context.cli[s.dest] != getattr(prev, s.dest):
+            if s.dest in context.cli and context.cli[s.dest] != history[s.dest]:
                 logging.warning(
                     f"Ignoring {s.long} from the CLI. Run with -f to rebuild the image with this setting"
                 )
 
-            if hasattr(prev, s.dest):
-                context.cli[s.dest] = getattr(prev, s.dest)
-                if s.dest in context.config:
-                    del context.config[s.dest]
-
-        context.only_sections = ("Include", "Runtime", "Host")
-    else:
-        context.only_sections = tuple(only_sections)
-        subimages = []
-        prev = None
+        context.cli |= history
 
-    if not in_sandbox() and have_history(args, tools=True):
-        tools = Config.from_json(json.loads(Path(".mkosi-private/history/tools.json").read_text()))
-    else:
-        tools = None
+    cli = copy.deepcopy(context.cli)
 
     # One of the specifiers needs access to the directory, so make sure it is available.
     context.config["directory"] = args.directory
@@ -5085,7 +5118,14 @@ def parse_config(
 
     context.config["files"] = []
 
-    configdir = get_configdir(args)
+    configdir = finalize_configdir(args)
+
+    if (
+        (args.auto_bump and args.verb.needs_build())
+        or args.verb == Verb.bump
+        and context.cli.get("image_version") is not None
+    ):
+        context.cli["image_version"] = bump_image_version(configdir)
 
     # Parse the global configuration unless the user explicitly asked us not to.
     if args.directory is not None:
@@ -5094,9 +5134,11 @@ def parse_config(
 
     config = context.finalize()
 
-    if prev:
-        return args, tools, (*subimages, Config.from_dict(config))
+    if config["history"] and want_new_history(args):
+        Path(".mkosi-private/history").mkdir(parents=True, exist_ok=True)
+        Path(".mkosi-private/history/latest.json").write_text(dump_json(Config.to_partial_dict(cli)))
 
+    tools = None
     if config.get("tools_tree") == Path("default"):
         if in_sandbox():
             config["tools_tree"] = Path(os.environ["MKOSI_DEFAULT_TOOLS_TREE_PATH"])
@@ -5510,6 +5552,10 @@ class JsonEncoder(json.JSONEncoder):
         return super().default(o)
 
 
+def dump_json(dict: dict[str, Any], indent: Optional[int] = 4) -> str:
+    return json.dumps(dict, cls=JsonEncoder, indent=indent, sort_keys=True)
+
+
 E = TypeVar("E", bound=StrEnum)
 
 
diff --git a/mkosi/installer/__init__.py b/mkosi/installer/__init__.py
index 5a49da68..fa7418ae 100644
--- a/mkosi/installer/__init__.py
+++ b/mkosi/installer/__init__.py
@@ -22,7 +22,7 @@ class PackageManager:
         return Path("custom")
 
     @classmethod
-    def cache_subdirs(cls, cache: Path) -> list[Path]:
+    def package_subdirs(cls, cache: Path) -> list[Path]:
         return []
 
     @classmethod
@@ -85,7 +85,7 @@ class PackageManager:
             # configured package cache directory in this scenario, we mount in the relevant directories from
             # the configured package cache directory.
             if d == "cache" and context.metadata_dir != context.config.package_cache_dir_or_default():
-                caches = context.config.distribution.package_manager(context.config).cache_subdirs(src)
+                caches = context.config.distribution.package_manager(context.config).package_subdirs(src)
                 mounts += flatten(
                     (
                         "--bind",
diff --git a/mkosi/installer/apt.py b/mkosi/installer/apt.py
index becb6d38..1d78f22c 100644
--- a/mkosi/installer/apt.py
+++ b/mkosi/installer/apt.py
@@ -54,9 +54,13 @@ class Apt(PackageManager):
         return Path("apt")
 
     @classmethod
-    def cache_subdirs(cls, cache: Path) -> list[Path]:
+    def package_subdirs(cls, cache: Path) -> list[Path]:
         return [cache / "archives"]
 
+    @classmethod
+    def state_subdirs(cls, state: Path) -> list[Path]:
+        return [state / "lists"]
+
     @classmethod
     def dpkg_cmd(cls, command: str) -> list[PathString]:
         return [
@@ -97,6 +101,10 @@ class Apt(PackageManager):
             "mkosi-reinstall": ["apt-get", "install", "--reinstall"],
         }  # fmt: skip
 
+    @classmethod
+    def options(cls, *, root: PathString, apivfs: bool = True) -> list[PathString]:
+        return super().options(root=root, apivfs=apivfs) + ["--dir", "/var/lib/apt/lists/partial"]
+
     @classmethod
     def setup(cls, context: Context, repositories: Sequence[AptRepository]) -> None:
         (context.sandbox_tree / "etc/apt").mkdir(exist_ok=True, parents=True)
@@ -167,7 +175,8 @@ class Apt(PackageManager):
             "-o", "Acquire::AllowReleaseInfoChange=true",
             "-o", "Acquire::Check-Valid-Until=false",
             "-o", "Dir::Cache=/var/cache/apt",
-            "-o", "Dir::State=/var/lib/apt",
+            "-o", "Dir::State=/buildroot/var/lib/apt",
+            "-o", "Dir::State::lists=/var/lib/apt/lists/",
             "-o", "Dir::Log=/var/log/apt",
             "-o", "Dir::State::Status=/buildroot/var/lib/dpkg/status",
             "-o", f"Dir::Bin::DPkg={context.config.find_binary('dpkg')}",
diff --git a/mkosi/installer/dnf.py b/mkosi/installer/dnf.py
index dec41293..82d23a36 100644
--- a/mkosi/installer/dnf.py
+++ b/mkosi/installer/dnf.py
@@ -25,7 +25,7 @@ class Dnf(PackageManager):
         return Path("libdnf5" if cls.executable(config) == "dnf5" else "dnf")
 
     @classmethod
-    def cache_subdirs(cls, cache: Path) -> list[Path]:
+    def package_subdirs(cls, cache: Path) -> list[Path]:
         return [
             p / "packages" for p in cache.iterdir() if p.is_dir() and "-" in p.name and "mkosi" not in p.name
         ]
@@ -148,10 +148,10 @@ class Dnf(PackageManager):
             "--setopt=keepcache=1",
             "--setopt=logdir=/var/log",
             f"--setopt=cachedir=/var/cache/{cls.subdir(context.config)}",
-            f"--setopt=persistdir=/var/lib/{cls.subdir(context.config)}",
             f"--setopt=install_weak_deps={int(context.config.with_recommends)}",
             "--setopt=check_config_file_age=0",
             "--disable-plugin=*" if dnf == "dnf5" else "--disableplugin=*",
+            "--setopt=persistdir=/buildroot/var/lib/dnf",
         ]  # fmt: skip
 
         for plugin in ("builddep", "versionlock", "reflink"):
diff --git a/mkosi/installer/pacman.py b/mkosi/installer/pacman.py
index 08d8e0f5..55b34f0f 100644
--- a/mkosi/installer/pacman.py
+++ b/mkosi/installer/pacman.py
@@ -35,12 +35,12 @@ class Pacman(PackageManager):
         return Path("pacman")
 
     @classmethod
-    def cache_subdirs(cls, cache: Path) -> list[Path]:
+    def package_subdirs(cls, cache: Path) -> list[Path]:
         return [cache / "pkg"]
 
     @classmethod
     def state_subdirs(cls, state: Path) -> list[Path]:
-        return [state / "local"]
+        return [state / "sync"]
 
     @classmethod
     def scripts(cls, context: Context) -> dict[str, list[PathString]]:
diff --git a/mkosi/installer/zypper.py b/mkosi/installer/zypper.py
index f36c8af3..0f6c689b 100644
--- a/mkosi/installer/zypper.py
+++ b/mkosi/installer/zypper.py
@@ -23,7 +23,7 @@ class Zypper(PackageManager):
         return Path("zypp")
 
     @classmethod
-    def cache_subdirs(cls, cache: Path) -> list[Path]:
+    def package_subdirs(cls, cache: Path) -> list[Path]:
         return [cache / "packages"]
 
     @classmethod
diff --git a/mkosi/resources/man/mkosi.1.md b/mkosi/resources/man/mkosi.1.md
index c42f77ab..20914f55 100644
--- a/mkosi/resources/man/mkosi.1.md
+++ b/mkosi/resources/man/mkosi.1.md
@@ -178,7 +178,11 @@ The following command line verbs are known:
     simple versioning scheme: each time this verb is called the version is
     bumped in preparation for the subsequent build. Note that
     `--auto-bump`/`-B` may be used to automatically bump the version
-    after each successful build.
+    as part of a build. The new version is only written to
+    `mkosi.version` if the build succeeds in that case.
+
+    If `mkosi.bump` exists, it is invoked to generate the new version to
+    be used instead of using mkosi's own logic.
 
 `genkey`
 :   Generate a pair of SecureBoot keys for usage with the
@@ -281,11 +285,14 @@ Those settings cannot be configured in the configuration files.
     Defaults to two years (730 days).
 
 `--auto-bump=`, `-B`
-:   If specified, after each successful build the version is bumped in a
-    fashion equivalent to the `bump` verb, in preparation for the next
-    build. This is useful for simple, linear version management: each
-    build in a series will have a version number one higher then the
-    previous one.
+:   If specified, the version is bumped and if the build succeeds, the
+    version is written to `mkosi.version` in a fashion equivalent to the
+    `bump` verb. This is useful for simple, linear version management:
+    each build in a series will have a version number one higher then
+    the previous one.
+
+    If `mkosi.bump` exists, it is invoked to generate the new version to
+    be used instead of using mkosi's own logic.
 
 `--doc-format`
 :   The format to show the documentation in. Supports the values `markdown`,
@@ -307,8 +314,8 @@ Those settings cannot be configured in the configuration files.
 `--rerun-build-scripts`, `-R`
 :   Rerun build scripts. Requires the `Incremental=` option to be
     enabled and the image to have been built once already. If `History=`
-    is enabled, the config from the previous build will be reused and no
-    new history will be written.
+    is enabled, the history from the previous build will be reused and
+    no new history will be written.
 
 ## Supported output formats
 
@@ -1583,15 +1590,11 @@ boolean argument: either `1`, `yes`, or `true` to enable, or `0`, `no`,
     up in the generated XFS filesystem.
 
 `History=`, `--history=`
-:   Takes a boolean. If enabled, **mkosi** will write information about
-    the latest build to the `.mkosi-private` subdirectory in the
-    directory from which it was invoked. This information is then used
-    to restore the config of the latest build when running any verb that
-    does not rebuild the image or when running any verb that may rebuild
-    the image without specifying `--force`.
-
-    Note that configure scripts will not be executed if we reuse the
-    history from a previous build.
+:   Takes a boolean. If enabled, **mkosi** will write the configuration
+    provided via the CLI for the latest build to the `.mkosi-private`
+    subdirectory in the directory from which it was invoked. These
+    arguments are then reused as long as the image is not rebuilt to
+    avoid having to specify them over and over again.
 
     To give an example of why this is useful, if you run
     `mkosi -O my-custom-output-dir -f` followed by `mkosi vm`, **mkosi**


Test patch:
diff --git a/tests/test_json.py b/tests/test_json.py
index 66f56112..3d402f23 100644
--- a/tests/test_json.py
+++ b/tests/test_json.py
@@ -41,6 +41,7 @@ from mkosi.config import (
     Verity,
     Vmm,
     VsockCID,
+    dump_json,
 )
 from mkosi.distributions import Distribution
 
@@ -92,7 +93,7 @@ def test_args(path: Optional[Path]) -> None:
         wipe_build_dir=True,
     )
 
-    assert args.to_json(indent=4, sort_keys=True) == dump.rstrip()
+    assert dump_json(args.to_dict()) == dump.rstrip()
     assert Args.from_json(dump) == args
 
 
@@ -615,5 +616,5 @@ def test_config() -> None:
         workspace_dir=Path("/cwd"),
     )
 
-    assert args.to_json() == dump.rstrip()
+    assert dump_json(args.to_dict()) == dump.rstrip()
     assert Config.from_json(dump) == args