Change:
The systemd service can prevent from succeeding: This service runs , and the man page says: I am not familiar with this tool, but my experience is that after my laptop has been up for some time (e.g. days), will often return an error code rather than correctly determine the network is up, thus killing any future commands. Steps to reproduce the behavior: your machine in a state where does not return success (not sure how to do this on demand). failure. should not fail due to an erroneous network check. This is tricky as it is not a issue per se but rather an issue with a presumably flaky systemd service. It is easy enough to disable this service manually: And perhaps this is the best solution. But a number of my coworkers all ran into this issue independently, so I thought it merited an issue for discoverability, if nothing else. My gut reaction is that a flaky check should probably not be required by default, but I don't know enough about this service's importance/fragility to say. This issue was noticed only recently, both on and . Please run and paste the result. Thanks!
Possibly it relates to this? Edit: Nevermind, that commit is not in .
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
This is related to udev not initializing devices. NetworkManager never completes startup because a WireGuard interface is never initialized by udev. A workaround is just putting the affected device into .
I'm wondering if there is any solution to this. Since this triggers for me always. So, I'm not even sure if my does actually complete. As mentioned in the initial description of the issue: This never seems to be the case, since the actual message is "Started Network Manager.". Apart from that I'm wondering why it relies on a log message string, when the status of would be way less prone to errors. EDIT: no idea why, but it went away ... Everything works as expected again.
I think I'm running into this. It only happens after I've connected my USB-C dock with an ethernet connection at least once after boot. (Note: I'm running tmpfs on root, so my system should "forget" everything about my dock on reboot.) I'm also running , not .
don't you dare!
Any chance you can help with this?
I don't use NetworkManager so I wouldn't know, but in the case of systemd-networkd there are relevant options under : . I recommend at least setting the former to on laptops. Warning about failed units is pretty much that the activation script does, so it's probably fine (but the failure should be fixed, of course).
BTW I've "fixed" this by setting But this is really an upstream systemd bug.
My Ubuntu has: My latest NixOS (22.05) config has: based on : Where is this coming from in your configurations? Also:
The reference to comes from upstream service unit , not from nixpkgs itself.
I'd vote for disabling this service until we can make it reliable. It's doing no good currently.
I've been tripping over this bug for quite some time now and it is annoying for users. As mentioned above, the error can be worked around with: I was concernd if there might be other dependencies or services that require this to be enabled, so I grepped through nixpkgs for both. These are the mentions: The actual definition of this service can probably be removed Service devinition Used in various nixos tests, which should not be relevant to normal operation of the system (?) nixos/tests/systemd-networkd-dhcpserver-static- nixos/tests/systemd- nixos/tests/systemd- nixos/tests/systemd-networkd- Upon first look the usage of these two services seem minimal to me and they are causing more problems that doing good. Agreeing with proposal, I'd vote to disable them per default. If this is agreed upon, I can submit a PR
I've been running with that service disabled for 6 months now and have not experienced a single issue. Don't count my voice too heavily, though :wink: ! :+1:
If we're going to work around this I'd still prefer as the other workaround just masks the issue while udev's still half-broken.
For some reason, that didn't work for me. On rebuild it said "not restarting service"
Yep confirming what said: Still seeing the issue on HEAD. Disabling wait-online like mentioned previously fixes nixos-rebuild.
is a dependency of : As others pointed out, the (aka. ) flag is causing some deadlock: I guess using is a more accurate notification than simply waiting for to be up, and I don't know why it sometimes fails to notice that the startup is complete, nor whether removing it has any significant drawbacks, but keeping it causes to fail often enough to bother me, so until a more correct fix is available, I'm personally removing it:
Also running into these deadlocks on nixpkgs-unstable. edit: see mentioned below that this might be related to wireguard. I use tailscale - might try to play around with whether removing that setup makes any difference on this issue for me.
This has been happening to me consistently for over a year. The cause seems to be due to what I've described in I used (a simplified version of) workaround, but using the workaround from seems to be even better. However, I don't know how to include that in my system configuration rather than running it manually when needed.
My instance of this issue is indeed caused by a wireguard interface, but adding the interface to (via name or ) did not change a thing. Having done some more digging online, I believe I know why: prior to udev initialization completing for a link, NM cannot be sure what the name or type of the link will be (as udev may make changes), and so it is not possible to filter a device prior to udev marking it as configured. I'm still looking for a workaround.
I have the issue on the MacBook Pro 2019 16,1 with t2 patches after switching to iwd, because of network issues. worked for me.
I'm new to NixOS, I have this error as well: I put this line in my (), but then I get this error: Right now I can't update anymore, so I am stuck and a fairly recent clean NixOS install. Is there a way to force the update for now? What is the best structural solution? Thank you!
Looks like adds this in the beginning of whereas most people eventually modify that to , providing with the missing argument. I think nixos-generate-config should be updated to include by default. Although the default config doesn't need/use , it quickly becomes something new NixOS users want to use.
Thanx for the quick reply. I have { config, pkgs, lib, ... }: In the first line of my , but I get the same error...
Then please open a new issue or post on asking for help.
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
Ok, this solved it for me: Just did and rebooted. I left the lib in the . Thanx for the support!
So, to summarise, the actual error seems to be that NM waits for startup of some interfaces it thinks it should manage but they're never communicated as up to NM because NM was never supposed manage that interface? When I say , NM thinks it does not manage my tailscale0 interface: Why would it be waiting on it then? The other interfaces are up and I'm sending you this message via one of them, so it can't be waiting on those. I'll try to explicitly add it to and report back after a few rebuilds in a few weeks.
No, is just weirdly broken and locks up if you try to do it on an already running system. It's probably an easy code fix but I'm not familiar with the codebase at all.
Is that something that has to be fixed upstream? If so, could you create an issue for it?
There's already an issue (kind of):
Reading through those. Are we using the mentioned flag? If so, maybe we can remove it? Running manually hangs for me, while (without ) seems to work correctly.
We are, and we have to, at least on initial boot.
Alright, I'll stick with my workaround then for now. I would like to contribute a real fix for this or sort it out properly, but don't know how to continue.
I had this problem for well over a year, but since I this to my the problem has not reoccurred: Nor have I had any other network problems. Also, seems to run at various times that aren't related to reboots or
what does that exactly do? If I understand the man page correctly, only omits the output, without changing any functionality
The was already there, and I'm just removing the . Sorry I didn't make that clearer. With the extra configuration:
Adding to on my system that's currently experiencing this issue did not resolve the problem.
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
Is there any way we can properly fix this issue ?
Not on the NixOS side; we'd have to fix it upstream I imagine.
I'll try and see how I can raise this issue upstream, we have to be careful to explain how this issue impact NixOS users but how it might benefit from fixing this issue (if there are any for others distros)
Is the problem that with on an already-up system hangs (what this thread seems to think)? Or is the problem that should not have (what seems to think)? It depends on what you think "wait for X" should mean when "X" has already happened. Should wait for the next X, or should it return immediately (no waiting necessary)?
I do not see any confusion about this from the upstream side; they did not claim that distros should not use at any point in the thread you linked. We want a flag that does what is currently documented to supposedly do: The manual even explicitly clarifies behaviour after bootup is complete. This is precisely what we want but simply does not work as documented. Instead of returning immediately, it blocks indefinitely under certain conditions.
<sIs this related to kernel/firmware upgrade? This problem just occurred on my machine after a , manually invoking also waited 30s and timed out, but after a reboot I can no longer reproduce this, even with another . I updated my kernel and firmware in the first , updating from linux 6.9.1 to 6.9.2.</sOh, NetworkManager itself is updated and restarted.
No, it's not.
I noted that is restarted before and . Also confirmed in journalctl: <details</detailsAnd if I before , will be started after is started, and then will exit gracefully.
Big if true. That feels like something that should be solvable using a few systemd Before/After directives in the right places. Though tailscaled.service already contains IIRC should be applied in reverse when stopping/restarting.
for me instead of did the trick..
could you please challenge upstream about the flag and its documented behavior ? EDIT : This seems to have been done by I believe Sorry for the tag
Just to add a +1 to the tailscale discussion - I did not have this issue at all prior to installing tailscale, and now have it any time a rebuild needs to restart NetworkManager.
~It seems currently doesn't have such mechanisms, neither does . For the Perl script, it just s and then s. Meanwhile the Rust version seems to be all threads to finish before starting the threads.~ ~A probable solution to this is to use combine these two commands into one , as stopped services will just get started on , and systemctl/systemd should be able (?) to resolve these Before/After relations if we invoke the command in one go. That doesn't work. I don't know how to edit the Perl script. For the rust version, maybe removing that and changing accordingly will be sufficient.~ ~—— in case this is really the cause of the problem.~ It seems I've made an over-complicated assumption.
Thank you for this observation - adding seem to have solved the issue for me
I'd invite everyone to try this out on their machines and report back. If this works, that'd be a quick and easy one-line patch to the tailscale NixOS module; fixing this for everyone.
I'm not sure it really fixes the underlying problem ( misbehaviour in the presence of the tailscale network device), but certainly stops nixos-rebuild hanging for me.
Can confirm this fixes it. +1 for including the fix in the tailscale module
Opened a PR: Testing and feedback appreciated
Is this only fixed when using tailscaled, for users not using tailscaled, they should apply which of the dozens workaround shared here ? Is this really a good solution for new users? Shouldn't something be also in NetworkManager's module ? Quoting previous comments :
have you observed this without using Tailscale? As far as I understand the issue is tied down to the combination of Tailscale firing up before NetworkManager and depending on it being online, ending up in a deadlock essentially
The same issue happened to me with wireguard in the past. I'll try the suggested fix
I have yes and I believe I'm not the only one
Mind posting your in a gist? We should be able to create a minimal flake and reproduce this with build-vm. Maybe cut out the VM's network access on purpose. We should reopen this issue then if this happens outside of just Tailscale & NM. EDIT: do you use wireguard? Tailscale uses wireguard so that could be the lower common denominator
I do use Wireguard yes, so it could be I have the same issue as Regarding my , it is all over the place currently, a minimal flake would be great but I can't provide my whole config at the moment, if I have time I'll try in a VM with a single Wireguard interface setup.
Can anyone confirm whether they still experience this issue without using tailscaled?
As recently as two years ago, I had this issue without knowingly using tailscaled. (Posted to .) Since then, I'm using the workaround from the OP and have lived happily ever since. The problem is perhaps that tailscaled is not the only service that activates , so it doesn't make sense to fix this issue in tailscaled. Perhaps this issue has since been fixed by an unrelated patch upstream (however that would happen), but I'm not optimistic. Multiple people (for example , linking to ) have pointed that this is probably an upstream bug in systemd/udev. Let me try to get rid of the workaround and see if it is still an issue.
Thanks I may try to reach out to systemd/udev maintainers about issue hopefully we can nail this issue down once and for all
We did receive an answer from Poettering but I don't know how to properly answer back, if anyone more experienced could pitch in :

Discussion:
environment.systemPackages = [ cfg.package ]; # for the CLI systemd.packages = [ cfg.package ]; systemd.services.tailscaled = { <ins> after = lib.mkIf (config.networking.networkmanager.enable) [ "NetworkManager-wait-online.service" ]; </ins> wantedBy = [ "multi-user.target" ]; path = [ pkgs.procps     # for collecting running services (opt-in feature)