Change:
When I set , doesn't get mounted nor does it get populated with the suid wrappers. This makes , and unusable. I'm not sure if anything else from the activation script is not working / running either. All I did was enable the mentioned systemd in initrd option on a system running NixOS 23.05, run and then reboot. I expected to be mounted, to have a symlink and for , and to work. My NixOS configuration is extensive/complex and I do have local modifications in , so it's not inconceivable that it is related to something I'm not doing right. cc Not sure if I should cc someone else, perhaps knows.
That's very odd. I certainly have on all my systems with systemd-initrd. One thing to note is that we move and activation scripts so they run in a chroot during stage 1, rather than running just before systemd in stage 2. The reason for this is so that can pass systemd state from stage 1 to stage 2, but it requires stage 2 init to be the actual systemd binary instead of a wrapper script like it is without systemd-initrd. Point being, if you're using activation scripts or (which (effectively) includes ) to do anything fancy, those may be experiencing slight confusion for running in a chroot? We tried to make it compatible though.
On this system, I am not using or but I was using this activation script: I tried disabling this activation script and it sems to have solved the mounting issue but it somehow introduced different, perhaps even more serious problems when booting: ! Apparently, I couldn't even login as root. It's quite baffling how removing the activation script can cause these problems, so I'm now wondering if it was just a coincidence and in fact it's an unrelated problem. But when I pressed , the boot process seemed to continue fine and was even mounted automatically. Unfortunately, I have disabled 's normal operation. Instead, I configured to forward messages to (for which I have greatly increased its buffer size) and any such messages during boot should make it to my log files when rsyslog starts, but for some reason I don't usually see many NixOS stage1-related messages in my logs, including any error messages related to this boot failure. That said, I don't know whether my configuration settings are being properly set when using systemd in initrd.
Ok, so I have booted this configuration (without the activation script) two more times, and it seems that the activation script was a red herring: I am back to the same issue as in the initial description. So apparently the activation script that I had has nothing to do with the mounting issue. I'm guessing this also means that the boot failure I experienced before is probabilistic and apparently, not easy to reproduce given that I only experienced this failure once in about 10 boots total that I've done so far with systemd in initrd. By the way, even though is not mounted automatically, I am able to manually fix the issue by logging in as root and then running , but of course, this is not really what a user should have to do.
I'm attaching a log of my boot process, but please note that the timestamps are highly misleading, i.e. the order of events should be correct but the time between events is not: When I compare it to the log in , in which I had the same configuration but actually did mount successfully, here's what jumps out the most for me: the boot where was not mounted properly, the service runs before gets unmounted. the boot where was mounted properly, the service runs after gets unmounted. I'm not sure if this is the root cause, as I am out of my depth here, but maybe it helps.
I believe the timestamps would be right if you used journald instead of rsyslogd.
Sure, and I really like journald's features and admin interface, but when I used it a few years ago, its implementation was extremely unreliable, so I simply cannot use it. In other words, I really need my log daemon to be reliable if I want to have any hope of diagnosing hard-to-debug issues like the ones we've been discussing or many others that I run into over the years. Here are some bugs in journald that (at least a few years ago) frequently caused me to lose a large amount of log messages: was not able to properly handle jumps in the system clock. This was especially problematic in my Raspberry Pis, which don't have a battery-backed clock and therefore always boot up with the time set to the year 1970 (or whatever it is), until has a chance to connect to an NTP server and set the clock correctly. This would actually lead me to losing entire days worth of log messages (in my Raspberry Pis, at least), for no good reason. This also caused many bugs in the utility, as it wasn't able to track different boots correctly and many other similar issues. systemd also has had some bugs with jumps in the system clock, but they are/were far less severe -- I think the worst that happens is that some timer units are started when they clearly shouldn't be. lost log messages when a service was starting, immediately sent some messages to stderr and then immediately failed. This made debugging some service failures much harder, as it is not even clear to the user when this happens -- the error messages were simply lost and there was no way to retrieve them nor was there any indication that these messages were lost. If I recall correctly, the journald developers even said they weren't going to fix this issue, although I could be remembering incorrectly (this was a few years ago). was not able to handle power loss without corrupting its log files (?) I don't necessarily mind that journald uses a binary format to store its logs, although I wish there was a simple tool to read them, e.g. when booted from a rescue system (perhaps journald or journalctl can do that?). But to me it's unacceptable that the logs corrupt themselves just because of an abrupt power down. To be clear, this may actually not be an issue with journald's design: it's impossible to distinguish between an abrupt power loss and a maliciously (or accidentally) truncated log file, and it's useful to know when this happens. However, even if that's the case and hopefully (!!!) no messages are being lost unnecessarily (due to journald being unable to fully parse a truncated log correctly), the user interface should definitely be improved: there should be a clear distinction between a log that was simply truncated and a log that was corrupted in any other way, as the latter is a much more serious issue than the former, since it points to a definite problem, not something that could be happening day-to-day. There was also another unrelated problem that led to journald corrupting its log files very frequently on some of my machines, but I think this may have been a bug in ZFS's mmap implementation that was fixed recently. I only experienced this corruption with journald, even when there were no abrupt power downs, but I think it's because journald made heavy use of mmap for its log files in strange ways instead of doing I/O normally (although this is also a red flag for me, as I think mmap precludes handling I/O failures gracefully, at least in some cases). In short, I wouldn't mind using journald but they'd definitely have to fix the above issues first, especially those that lead to message loss.
We're not here to debate the pros and cons of journald in all contexts. I'm not asking you to switch your systems to journald. I'm telling you we'd get better logs in this particular bug report if journald were used. That matters a lot because right now it's very hard to help with how little info we have, including the fact that you haven't shared almost any of your configuration.
Sorry, I misinterpreted you. I thought you were suggesting me to switch to journald. The NixOS configuration for this system (my laptop) is 3.5K lines of code (excluding blank lines and comments), plus I have about 200 patches applied to , so I'm not sure sharing all of that information would necessarily be productive nor would I like to do that, for privacy reasons. If you'd like me to provide some specific details or at least narrow it down to a few specific options or even an option tree, or run any commands, or change anything, I could try to do that instead. Or if you could provide a comment similar to the one you provided in then I can try looking into those details to see if I can debug the issue myself, or at least see if I can get more relevant information. As an example, right now I am in the process of trying to extract my initrd and look into the configuration of the units you mentioned in that issue to see if anything looks suspicious. In any case, if you don't want or you don't think you can help me this way, that's also perfectly fine! You have no obligation to help debug my issues, some of which may be unique to my configuration. In any case, I really appreciate and am very grateful for all the help you've provided so far! Thank you.
I was suggesting that you try it for a couple of boots just to get the more useful logs. The ordering of systemd-related events is very very important in this context and I expect journald to order those most correctly, especially if you're saying rsyslogd can't get the timestamps right. Your entire 2Kloc config isn't necessary, but the bits that have anything to do with initrd would be very important to this issue. Anything at all that affects the generation of initrd, the activation scripts, or the should pretty much cover anything we'd need to see to understand an initrd failure. Like if you have any style options, then consider that those don't work in systemd stage 1:
As I said before in this issue, the ordering between events is correct in my logs. These log messages are sent by journald in the correct order to , which then copies to in the same order. However, I configured to ignore the timestamps due to an unrelated issue, so the timestamp that gets applied to these boot messages is the timestamp when copied each individual message from to , not the timestamp that was saved when the message was stored in . So in this case, you cannot rely on the elapsed time between events in my logs (for those events that happen before starts), but you should be able to rely on how the messages are ordered between themselves. I could try using if the elapsed time between events would be useful information to have. Ok, let's see. I don't know exactly what affects the generation of initrd, but here are all the options that I have set in this system: I don't have any other options. But I have , not sure if it's relevant. Hopefully it's not deleting anything it shouldn't in stage1. In this system I believe I only have one manually activation script (the one I showed above), but I tried disabling it and the issue persisted. I don't have any , or in my config. I don't think I have any patches in that could be related to these issues, except for the patches you provided in (including the dependency, which I haven't reverted yet). ... that is, unless the Network Manager services are started in initrd, which I don't think is the case? I'm not aware of anything else in my system that could be relevant to this issue or ...
Maybe some logs with on the cmdline would help. As described so far, the ordering of systemd events makes absolutely no sense.
Here are the requested logs: Thanks!
I've checked the logs and I've noticed the following events happen in order: seems to run successfully. Presumably, this is when gets mounted inside the chroot. , starts get unmounted, along with other mountpoints under , the "Switch Root" service is started So it seems that gets unmounted just before switching the root and then the activation script does not run again, so it never gets remounted afterwards. Presumably, got unmounted due to the command, which stops all the mount units that are not dependencies of the switch-root target?
If this were a problem, no configuration with systemd initrd would work. But in fact, has , which keeps FSes around upon .
But does not contain any dependencies on the unit. This is what the man page says: So, the dependency between the unit and the unit is supposed to be by . But as you can see from the logs, does not find the filesystem. In fact, if you search for in the latest debug log that I posted, only this is found: (And I think even this entry is wrong because I think it should be , not ).
While you have a good point there, it doesn't explain two things. does no one else have this problem? is everything else also being unmounted? That is positively antithetical to the dependency relationship between and . No, it appends that implicitly correctly.
I don't know exactly, but if you add as a kernel command line parameter and post your logs, we could look and see why this doesn't happen to you. But I can tell you that the only two times that my system did not unmount when I enabled systemd in initrd, it was because some filesystems under could not be unmounted due to them being busy: So perhaps your boot is working correctly by accident, not because everything is configured properly. Or perhaps your configuration is different from mine in some important way. But we won't know until we can compare our logs. I don't understand this question. As far as I can see, they are being unmounted because cannot find any entries in either nor , except for the root entry that I mentioned. Since cannot find any entries for any filesystems, the dependencies between their mount units and are not being . This explains why the command stops all these units even though is a dependency. It does not matter that is a dependency because the dependencies between and the the mount units do not exist, which is why the mount units are getting stopped.
The file contains the following comment, which provides more insight: As far as I can tell, gets called with as part of the . This does not find any filesystems except for the root fs, and even if it did, it wouldn't actually add any dependencies between the mount units and , but I think that's actually what it's supposed to do. However, it should trigger a daemon reload after has been mounted, which would then reinvoke as an actual unit (and dependency) generator, which would then find the filesystem entries in . In my system, I can confirm that the daemon reload is correctly triggered and that this happens after has been mounted. So far so good. However, I am not sure how is supposed to be reinvoked as an actual generator nor am I sure whether this is happening on my system.
Actually, I can confirm that runs and succeeds, along with other generators, before is reached. However, in this case I think it doesn't matter whether is being invoked, because is not part of so would never add a dependency between the unit and the unit. In fact is mounted only after the target is reached, as part of the . So I guess this is why is being unmounted? Or is there any other generator that is supposed to create this dependency? With this amount of obfuscation in systemd's implementation it's hard to tell... :)
Actually, the same issue with exists with . So now I have a system that has booted with systemd in initrd but has not been properly bind mounted in read-only mode. Are you able to run as root without any errors, or does it fail with a error? That command should fail with a read-only FS error (when ran as root)... it should not succeed.
Another question: do you have and in your ? If not, why would you expect to add these filesystems as dependencies of ?
defaults to true for mount units. They shouldn't be stopped by an command. EDIT: That's my bad, I should have pointed that out much earlier. I kinda forgot tbh :P
Can you try with stock nixpkgs? If there's a bug in your 200 patches, obviously we can't help with that. What we need is a minimal way to reproduce this issue.
Just experienced this issue (sudo not working) after updating to I'm also using ZFS EDIT: after reboot everything works fine. Not sure what actually happened the first time.
Ah, that's interesting, I didn't know that. That said, I also noticed that systemd is stopping itself right after starts. And of course, is a direct dependency of , so systemd shouldn't have done that, I think (well, according to systemd's documentation, which unfortunately is quite vague). To be honest I am starting to suspect that this is a bug in systemd itself, because I don't think any part of my configuration or patches should change the behavior of systemd or anything in the initrd (except perhaps for a couple of things I will mention at the bottom of this comment), since along with grub and the kernel, these are the very first things to run so they shouldn't be affected by anything that runs afterwards. And as far as I can tell, I haven't modified systemd or any systemd units that run inside the initrd (apart from the patches you provided and that I mentioned earlier). So I'm guessing, either there is a bug in systemd, or maybe there is some kind of dependency screw up (maybe some change done by NixOS?), but I find it unlikely given that there are so few units involved. Perhaps I will do an exhaustive manual check of all the dependencies just in case, when I have the time. FYI, the command has been a source of bugs in systemd and some of them are still unsolved. In fact, it seems to be worse because the systemd developers that the command is known to be broken and they have recommended not to use it. Also, similar bugs have been reported when there are unusual dependencies or dependencies that directly or indirectly lead to cycles. That said, I'd imagine that every Linux distro that uses systemd (almost all of them, basically) is using the same command during boot, as well as identical units in this part of the boot process, because as far as I can see, that's how systemd is configured to boot by default. Since this is how almost every Linux distro boots nowadays, it's extremely surprising that nobody else is running into this issue, which is the same point you had made earlier. But still, for the reasons I mentioned above, it wouldn't be impossible for the bug to be in systemd. Or at the very least, there is a very serious usability issue in systemd because even expert users and/or developers who (along with me, I suspect) have been using and administrating systemd-based systems and other Linux systems for years or decades cannot figure out why it's behaving so surprisingly. Furthermore, there doesn't seem to be an easy way to find out why it is behaving this way, i.e. which rules or events are directly triggering each subsequent event. We can try to speculate what the problem is based on the order in which the events are happening, but this can only take us so far and it's very hard to be sure or to investigate what are the actual causes of such surprising behavior, and whether it's a bug in systemd or whether systemd is correctly following the specified unit rules (of which there are many, of many types, which interact in very complex ways, and some of which are unobservable because they are generated on-demand during boot). Well, that would surely be nice and I can try, but I assure you that would be a lot of work. Due to all the bug fixes and a few modifications and/or features, my configuration doesn't evaluate properly in stock nixpkgs. Furthermore, even if it did, it wouldn't build correctly, because the majority of my patches are bug fixes for build failures that I've experienced in my particular build environment. That means I would literally have to go through almost all the patches one-by-one to see if I could eliminate them, along with any configuration that depends on it (if necessary). I think it would be much easier to start by manually bisecting my NixOS configuration, in the sense of trying to remove half of the config, see if the issue remains, and keep repeating that until I figure out which parts of the configuration trigger the issue. Only then it would make sense to start removing patches, because then very few of them would be necessary anymore. But that would still be a lot of work, because my NixOS config is extensive and complex, spread out over multiple interdependent files and I can't really change my system directly, I would first have to set up my config to boot in a VM, which will also not be trivial despite the built-in NixOS support for that. Furthermore, if the underlying bug turns out to be nondeterministic (e.g. due to some kind of race condition) and it only manifests itself under some set of circumstances, it could make it much harder to pinpoint exactly where the problem is, if the minimal config doesn't trigger it frequently. The comment and subsequent edit provided by adds convincing evidence that this is indeed a nondeterministic issue which may be hard or impossible to reproduce in a minimal config. If that is the case, it will be much easier to debug the issue in my current full config, which can reproduce it 100% of the time. But still, I will try doing that when I have the opportunity and see how far I can get. Finally, let me just mention a few things that come to mind regarding what could be affecting systemd or initrd in my particular configuration. As far as I can remember right now, the only non-standard parts of my config that could affect them are: 1) my use of a ZFS pool for my root fs (and another one for my fs), 2) my custom Linux kernel config, which apart from the standard NixOS options contains additional recommended hardening options, and a few others as well, 3) the use of gcc's as a build option for my entire system, and 4) using NixOS itself (which is quite different from other distros). All of these have been the source of problems/bugs in the past, some of them quite unusual and/or tricky, but nothing substantial that couldn't be figured out eventually. I also have one kernel patch but it's entirely unrelated, I believe, as it's just a workaround for a hardware bug (USB SATA adapter) that a kernel maintainer provided to me but didn't want to integrate upstream. I think everything else is probably being used by at least hundreds of thousands or even millions of other systems, even combined together, I would guess (such as using UEFI + grub + LUKS, for example). Sorry for my extremely verbose comment(s)!
For the sake of anyone who isn't subscribed to , I have that could be helpful in diagnosing this issue as well, since I think they are related and may even have the same root cause, as both of them are caused by systemd unexpectedly unmounting filesystems in the exact same part of the boot process.
I'm finally pretty sure that I've found the root cause of this issue. I'm finally posting this comment from a NixOS system that has booted correctly with systemd in initrd for the first time since I've been trying. That is, and have now been properly mounted during boot, finally. I don't think I could've figured this out without looking at the graph of systemd units and their dependencies/requirements, containing all the units in the initrd that are relevant. The graph of dependencies I'm referring to is similar to the ones I posted in . However, I've made the following improvements: of obtaining the graph from the emergency shell after has failed (due to the problem in ), I've obtained it from the emergency shell that started due to changing so that it contains instead of . This means that I've ensured that always fails and never calls . This also allows me to log into the emergency shell and get the graph of dependencies at just the right point in the boot process. In other words, this is a great improvement because it means that I've obtained the graph just before systemd was going to misbehave, which means that the filesystems are still mounted and we can inspect their dependencies properly to see which dependencies/requirements could cause them to be unmounted by the command. 've removed a lot of irrelevant clutter, by not only filtering out the edges representing ordering between units (it's clear that the ordering is not actually the problem, the problem is that the mount units are being stopped when they shouldn't), but I've also filtered out other irrelevant units like all those temporary mountpoints I've created so that I could run to obtain the graph of dependencies, all the conflict dependencies (it's clear that is not the problem, as we are booting rather than shutting down), all the emergency shell related dependencies (not really relevant either) and also all the conflicts. The conflicts I'm referring to is the clause that each mount unit seems to "invisibly" contain (I say invisibly because I can't see it with e.g. , but it can be seen with ). Anyway, I think these conflicts are not really relevant because AFAIU they only exist to ensure that if is started, then all the mount units are stopped (i.e. all the filesystems are unmounted). This may appear to be suspiciously related to the problem I'm experiencing in this issue, because all (or almost all) the filesystems are in fact being unmounted, but I don't think it's related because has no other dependency except for these conflicts clauses (apart from a dependency that I had removed, I suppose... IIRC exists because it's supposed to be started during shutdown). That said, besides the mount units, it does also conflict with , but I don't think that's problematic (?). Anyway, these edges were also cluttering the graph visualization and I don't think they are relevant. So here's the new graph visualization I've obtained after all these improvements: Now, even with this improved visualization, it's not immediately clear what the problem is. But after some analysis I think it's pretty clear what is going on. Here's what I think is happening during the boot process, in order: of all, I think everything works perfectly fine until we reach . , as mentioned multiple times, starts the command. So far so good. means that systemd was ordered to do the following: start stop everything that is not required or wanted by , or required/wanted by its requirements/wants, etc (recursively). (We are still good up to this point) "wants" to start, so systemd starts making preparations to start it. This is normal and has been part of the upstream systemd boot process for at least 7 years. conflicts with , so it cannot start until the latter is stopped, which means systemd starts making preparations to stop . This is also normal and has been part of the upstream systemd boot process for at least 4 years. , cannot stop immediately because requires it. This is when things start to go wrong. Since systemd is making preparations to stop , it now decides to stop , because the former cannot stop without the latter stopping first. is a required dependency of , which means that in order to stop , systemd has to unmount first. But of course, in order to unmount , systemd has to unmount all the filesystems inside first, including and . So basically, it appears that stopping is part of the normal, upstream systemd-in-initrd boot process, but since the ZFS root pool import service "requires" this udev service, then the service that imports the root pool is stopped as well, and this in turn causes all the mount units to be stopped as well (i.e. it causes the filesystems to be mounted). So if I'm diagnosing this issue correctly, it means that has exactly the same root cause as this one, and it also appears that this issue is 1) NixOS-specific, 2) only triggered if you enable the systemd-in-initrd option, and also 3) only triggered if you use a ZFS root filesystem (well, similar issues could happen as well if you have some non-root ZFS filesystems, like ). To test whether my theory was right, I applied this simple one-line patch to nixpkgs: This appears to have fixed the issue for me (I've made no other change). What this patch does is pretty obvious: it makes a "wanted" dependency of the services that import ZFS pools rather than a "required" dependency. This means that if possible, will still be started before the ZFS pool import services, but it also means two other important things: is stopped, there is no need to stop the ZFS pool import services. This means that the filesystems that depend on these ZFS import services will not be stopped as well (in a sort of chain reaction). This should fix the problem I was experiencing. I understand it, if fails, then the ZFS pool import services will be started anyway. This is a side effect of changing "requires" -"wants", but not necessarily what was intended to fix the problem. This may or may not be a problem (I really don't know). So, as you've probably figured out, I'm not entirely sure that my "fix" is correct, as there are other possible ways to fix the same issue. For example, as you can see from the graph of dependencies, an alternative way of fixing the problem would be to make "want" (instead of "require"), but this could have its own problems as well, as would start when fails, and would always be stopped before switching the root. So I'm hoping someone like could step in and see whether my fix is enough or if there's a better way to fix it. Anyway, I hope this is it, finally, and if so, thanks for everything and I'm really sorry for all the noise I've caused while trying to debug these two issues (including all my extremely long comments, many of which about irrelevant points)...
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
I too just hit this issue, on an upgrade between and , as noted in the above Discourse link.
Based on this comment, I decided to try rebooting the problematic generation again. Attempt 1: same as the first time, an apparently normal boot, but no wrappers, sudo, firejail etc. Attempt 2: systemd went to emergency mode failing to switch root, as in . I hit enter to let it continue, and it did, system came up, and everything is fine (this is what I'm running now). So there's definitely something racy involved. I'm going to look at differences between them. I don't have any custom changes to nixpkgs, and nothing particularly unusual in my config. Can also share logs if that's useful, but I'll see what else I can spot first.
Interesting: Both are a directory with just the zfs import service linked So, it looks like your workaround is basically undoing a change between these revisions (which I haven't yet looked to track down)
Some other relevant diffs in systemd generated config: At this point it is probably better to go looking for the source of these changes.. some of which are clearly from and probably not the issue, and others are less clear (I will continue with this in the morning)
I'm leaning more towards suspecting change being the correct fix, and basically being a missed part of the other change. Note that settle only 'wants', not 'requires', the udev service itself. This system happens to be one where I haven't yet migrated the partitioning away from grub+bpool to systemd-boot+ESP-only, and I note that there's a bunch of stuff in here related to bpool. I'm going to build an update for another system that has been migrated, and see how these dependencies fall out in that case. It might explain why hasn't seen this problem.
Yeah, without the bpool service, there's no chain of 'required' all the way from settle to , so (if I understand the above correctly) if that gets killed it no longer cascades to or others also getting killed. So I think the fix is to break that cascade at the start, rather than rely on the accident of no other pools being in the chain.
Oh, I should add, though I think it's probably obvious: I'm looking at the post-pivotroot stage2 systemd configuration. I think these two issues are the same bug, one in each stage, and affecting different mounts. I don't really know why/how the settle service might be getting stopped in stage 2, though. I definitely feel like there are probably some other mounts affected/missing too, this one just happens to be easy to notice. And, when the problem is viewed as a chain of 'requires' all the way from 'settle' to 'local-fs', it's clear why this is the missing part of the other fixes in the recent commit: we need to break the chain, but it belongs at the early end not the late end.

Discussion:
createImportService = { pool, systemd, force, prefix ? "" }: nameValuePair "zfs-import-${pool}" { description = "Import ZFS pool "${pool}""; <del> # we need systemd-udev-settle to ensure devices are available </del> <ins> # We wait for systemd-udev-settle to ensure devices are available, # but don't *require* it, because mounts shouldn't be killed if it's stopped. </ins> # In the future, hopefully someone will complete this: # https://github.com/zfsonlinux/zfs/pull/4943 <del> requires = [ "systemd-udev-settle.service" ]; </del> <ins> wants = [ "systemd-udev-settle.service" ]; </ins> after = [ "systemd-udev-settle.service" "systemd-modules-load.service"