Change:
# copy what we need.  Instead of using statically linked binaries, # we just copy what we need from Glibc and use patchelf to make it # work. <del> extraUtils = let # Use lvm2 without udev support, which is the same lvm2 we already have in the closure anyways lvm2 = pkgs.lvm2.override { udev = null; }; in pkgs.runCommandCC "extra-utils" </del> <ins> extraUtils = pkgs.runCommandCC "extra-utils" </ins> { nativeBuildInputs = [pkgs.buildPackages.nukeReferences]; allowedReferences = [ "out" ]; # prevent accidents like glibc being included in the initrd }

Discussion:
Hydra shows a kernel panic.
When I tested it locally it failed with exactly the same log tail.
Increasing the kernel loglevel to 8 reveals the following log: This shows that can't mount , because the module is not loaded. Comparing the the log section of the currently failing test run () with the last successful one () shows that the is missing the line. So failed to detect the encrypted device and this probably happened between ..
I am currently bisecting this, will post here once I have some results.
said this: Combining this with I'm not really sure suddenly stops creating the line. I am pretty certain I did check all the installation tests succeed. I'd be very happy if someone could double check my bisect. Apart from that, let's read some perl, I'd say :-)
I confirm the bisection: the test fails on and succeeds on the parent commit.
Can we always add into initrd, until we understand why doesn't add that line? I assume reverting the commti will be a bit tough, as it resulted in a lot of cleanups (and will be a world-rebuild, too)
appears to be the snippet that generates that bit of config. I'm not familiar with how that part of sysfs enumerates, but it seems like something that might be different when luks doesn't have udev
From
I did dig a bit further on the log line from Thanks for this! Turns out, after the test runs the command, a block device file with the same major and minor as is created in . Instead, should be a symlink to (according to from the package, which is present in . The error message we saw is udev tripping over the following line in there: udev doesn't want to replace the device file with a symlink, and just keeps things as-is (which causes the warning in dmesg). I tried manually changing back to a symlink, which got to properly detect things, being to initrd, and to succeed booting: I still don't fully understand how the device node is created (is it the binary, or some library code it's linked against (libdevmapper from lvm2), or another udev rule we don't want to have, or something else?) if this is intended behaviour.

I will check a bit how things look on NixOS systems before the systemd-cryptsetup commit, as well as other distros. If we can improve to still detect things properly if these are just device nodes, and not symlinks. i propose applying above patch to and to unblock channels, but we need to better understand and apply more fixes until the 20.09 release - might not create a bootable currently.
being a device instead of a symlink is very confusing. No idea why that would be happening From I deduce that can fall back to creating device nodes itself when support is disabled (which you do by setting ) but that's only in ; not in package in general... so bit confused
A little reproducer: On NixOS 20.03 this returns: a symlink as expected.
On my NixOS machine I get this: After all the LUKS containers are unlocked the system continues to boot as normal.
There are many other circular dependencies in NixOS (known issue for all distros, see ). Perhaps would it be possible to address it the same way we address dependency in and alike?
I don't follow. So far it doesn't seem that the circularity is really related to the problem we have here. And I have no idea what you mean by "address it the same way".
It's likely that you shouldn't listen to me. But, I got the slightest impression from the bisect that if lvm2 is not built with udev, it doesn't detect the encrypted devices as before. The idea I had in mind was no good, now I realise - we can't get a binary release of systemd and bootstrap it from there.
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
I found that taking one more step along the systemd → cryptsetup → lvm2 → udev=systemd dependency cycle before breaking the chain allows the test to pass.
Wow, amazing. Do other Linux distros don't care? because they don't distinct between a package's inputs and the package itself? For sure Guix hasn't encountered this issue since they don't even use systemd! See and .

Other distro's have been running into the exact same issue:
For reference, any cycle-breaking should also watch the issue addressed by , as that solution isn't really "foolproof", i.e. such changes might again lead to breakages that happen based on non-determinstic issues (based on filesystem ordering).
I believe I’ve now addressed that issue by simply reverting ; see my in .
Sounds OK at a quick look.
has been merged, cc so it can trickle through the next staging round.
Lets close this; and open a new issue on breaking the cycle in a less closure-size-bloating way (e.g. by separating libudev)
Why do we need to wait for staging? Is master even in a usable state without the patch? Is it that staging will be merged very soon anyways?
master works, except installations from a master-built install medium will fail to detect crypto volumes (which is why the installer tests fail and the channel doesn't advance). This will be backported into 20.09 before the release one way or another - we just don't want to merge it without hydra having built most of the packages - otherwise ZHF will get pretty annoying.
that has this change. Evaluation is ongoing. If tomorrow morning or around noon things look good it can be merged into master (and I suppose we branch off then).
This will likely be the case. However, I'm traveling
It's rebuilding stdenv ATM, so I don't think we'll have that many binaries around the noon yet.
This issue has been mentioned on NixOS Discourse. There might be relevant details there: