Change:
server = index: { pkgs, ... }: let <del> ip = builtins.elemAt allConsensusServerHosts index; </del> <ins> numConsensusServers = builtins.length allConsensusServerHosts; thisConsensusServerHost = builtins.elemAt allConsensusServerHosts index; ip = thisConsensusServerHost; # since we already use IPs to identify servers </ins> in { networking.interfaces.eth1.ipv4.addresses = pkgs.lib.mkOverride 0 [ <del> { address = builtins.elemAt allConsensusServerHosts index; prefixLength = 16; } </del> <ins> { address = ip; prefixLength = 16; } </ins> ]; networking.firewall = firewallSettings; services.consul = <del> let thisConsensusServerHost = builtins.elemAt allConsensusServerHosts index; in </del> assert builtins.elem thisConsensusServerHost allConsensusServerHosts; { enable = true; inherit webUi; extraConfig = defaultExtraConfig // { server = true; <del> bootstrap_expect = builtins.length allConsensusServerHosts; </del> <ins> bootstrap_expect = numConsensusServers; # Tell Consul that we never intend to drop below this many servers. # Ensures to not permanently lose consensus after temporary loss. # See https://github.com/hashicorp/consul/issues/8118#issuecomment-645330040 autopilot.min_quorum = numConsensusServers; </ins> retry_join = # If there's only 1 node in the network, we allow self-join; # otherwise, the node must not try to join itself, and join only the other servers. # See https://github.com/hashicorp/consul/issues/2868 <del> if builtins.length allConsensusServerHosts == 1 </del> <ins> if numConsensusServers == 1 </ins> then allConsensusServerHosts else builtins.filter (h: h != thisConsensusServerHost) allConsensusServerHosts; bind_addr = ip;

Discussion:
The I wrote is failing on (): The problem is that when machines reboot in quick succession, consensus (and cluster leadership) is lost permanently, and 2-of-3 servers cannot continue to serve requests. I have debugged it and found that it is already broken since (), but it was noticed only now. :red_circle: This is why we should run the relevant tests for every PR and should show big red warning signs when they fail! In any case, I have bisected the problem to be commit , introduced between Consul and . I have filed an upstream bug: I also found a way to make the test pass: But I don't think that's good, because I think this is a fundamental issue with Consul that affects operations (the 2-of-3 HA promise being broken), so I think it should be fundamentally fixed, and that the test is right. NixOS 20.03 still has Consul so it does not suffer from this bug, but the next NixOS release will. CC as previous consule package contributors.
IMO we shouldn't even use attr, but always refer to concrete version, like or . Seamless major consul upgrade is fine in docs, but disaster in reality (happened twice to me). That doesn't quite solve this particular problem, I know.
Yes, the key thing is that Consul doesn't permit downgrades of its state in . I would thus also always recommend to upgrade non-point-releases separately from NixOS upgrades.
Honestly I kinda wanna make a few changes to most of the Hashi packages and tests soon to provide more stable versioning (and reliable testing). I'll start cleaning it up this weekend hopefully
Upstream has a great explanation and recommended fix: Test fix and more tests in PR . Key change: . I recommend every Consul user to read that linked comment and the updated test to understand the upstream-recommended rolling reboot/upgrade procedure.