Change:
Describe the bug Please see for a more thorough investigation than this. ~I'm not sure I'd really classify this as a bug, but it is a least surprising behaviour on some docker storage backends, so I'd like to document this. I'm not sure whether docker really has any formal semantics with respect to the loading process.~ The TL;DR: Some s may be usable on some storage backends but not on others when symlinked paths are involved. To Reproduce Consider: This image fails to load on an ext4/overlayfs setup: But succeeds on a zfs-backed docker setup: I imagine this is because of the storage backend's behaviour wrt. earlier layers' files. Expected behavior Image loads the same on all backends. Additional context This was discovered because our docker images became unpullable on some hosts (overlay) but not others (zfs) after crate2nix started using symlinks for tested builds: might find this interesting. :smile_cat:
docker.. sounds like an upstream issue, no? What happens if you do the equivalent in a Dockerfile?
Yes, I tend to agree, both backends should behave the same, but I'm afraid "overlay" is the de facto standard, so the functioning version is probably the buggy one. I may report this upstream, but it's at least surprising that can produce unpullable images (never happened to me before!) :grinning: Uh, that would require a working memory on how to make Dockerfiles... Stand by. :stuckouttongue:
I can't seem to reproduce this with Dockerfiles. I suspect it does some logic to avoid having symlink breakage across layers, but I haven't checked the source. Maybe we should do the same.
Scratch the above, my assumptions were wrong. Here's what I now think is going on, seen only from the perspective of an overlayfs setup (forget about zfs, I've no idea what process the clone/snapshot mechanism uses to make this magically "work" and to what extent it actually "works") Given the snippet: We get an image which has a layer containing: This, I think, is the main culprit. If we go to the prior commit (the commented nixpkgs revision) the problem does not exist, and the corresponding layer looks like this: Which looks much more correct and doesn't have more than one entry for with one being a symlink (correct) and the other (implicitly) a directory. In other words, I believe this is a bug and was introduced in specifically in and relates to how disparate entries are somehow created because of some wonky symlink traversal. :smile_cat: cc

Discussion:
).strip() == "${if pkgs.system == "aarch64-linux" then "amd64" else "arm64"}" ) <ins> with subtest("buildLayeredImage doesn't dereference /nix/store symlink layers"): docker.succeed( "docker load --input='${examples.layeredStoreSymlink}'", "docker run --rm ${examples.layeredStoreSymlink.imageName} bash -c 'test -L ${examples.layeredStoreSymlink.passthru.symlink}'", "docker rmi ${examples.layeredStoreSymlink.imageName}", ) </ins> ''; })

contents = crossPkgs.hello; }; <ins> # layered image where a store path is itself a symlink layeredStoreSymlink = let target = pkgs.writeTextDir "dir/target" "Content doesn't matter."; symlink = pkgs.runCommandNoCC "symlink" {} "ln -s ${target} $out"; in pkgs.dockerTools.buildLayeredImage { name = "layeredstoresymlink"; tag = "latest"; contents = [ pkgs.bash symlink ]; } // { passthru = { inherit symlink; }; }; </ins> }

for path in paths: path = pathlib.Path(path) <del> files = itertools.chain([path], path.rglob("*")) </del> <ins> if path.is_symlink(): files = [path] else: files = itertools.chain([path], path.rglob("*")) </ins> for filename in sorted(files): ti = append_root(tar.gettarinfo(filename))