Change:
for m in machines: m.wait_for_unit("consul.service") <del> for m in machines: m.wait_until_succeeds("[ $(consul members | grep -o alive | wc -l) == 5 ]") </del> <ins> def wait_for_healthy_servers(): # See https://github.com/hashicorp/consul/issues/8118#issuecomment-645330040 # for why the `Voter` column of `list-peers` has that info. # TODO: The `grep true` relies on the fact that currently in #       the output like #           # consul operator raft list-peers #           Node     ID   Address           State     Voter  RaftProtocol #           server3  ...  192.168.1.3:8300  leader    true   3 #           server2  ...  192.168.1.2:8300  follower  true   3 #           server1  ...  192.168.1.1:8300  follower  false  3 #       `Voter`is the only boolean column. #       Change this to the more reliable way to be defined by #       https://github.com/hashicorp/consul/issues/8118 #       once that ticket is closed. for m in machines: m.wait_until_succeeds( "[ $(consul operator raft list-peers | grep true | wc -l) == 3 ]" ) def wait_for_all_machines_alive(): """ Note that Serf-"alive" does not mean "Raft"-healthy; see `wait_for_healthy_servers()` for that instead. """ for m in machines: m.wait_until_succeeds("[ $(consul members | grep -o alive | wc -l) == 5 ]") wait_for_healthy_servers() # Also wait for clients to be alive. wait_for_all_machines_alive() </ins> client1.succeed("consul kv put testkey 42") client2.succeed("[ $(consul kv get testkey) == 42 ]") <del> # Test that the cluster can tolearate failures of any single server: for server in servers: server.crash() # For each client, wait until they have connection again # using `kv get -recurse` before issuing commands. client1.wait_until_succeeds("consul kv get -recurse") client2.wait_until_succeeds("consul kv get -recurse") </del> <ins> def rolling_reboot_test(proper_rolling_procedure=True): """ Tests that the cluster can tolearate failures of any single server, following the recommended rolling upgrade procedure from https://www.consul.io/docs/upgrading#standard-upgrades. </ins> <del> # Do some consul actions while one server is down. client1.succeed("consul kv put testkey 43") client2.succeed("[ $(consul kv get testkey) == 43 ]") client2.succeed("consul kv delete testkey") </del> <ins> Optionally, `proper_rolling_procedure=False` can be given to wait only for each server to be back `Healthy`, not `Stable` in the Raft consensus, see Consul setting `ServerStabilizationTime` and https://github.com/hashicorp/consul/issues/8118#issuecomment-645330040. """ for server in servers: server.crash() # For each client, wait until they have connection again # using `kv get -recurse` before issuing commands. client1.wait_until_succeeds("consul kv get -recurse") client2.wait_until_succeeds("consul kv get -recurse") </ins> <del> # Restart crashed machine. server.start() </del> <ins> # Do some consul actions while one server is down. client1.succeed("consul kv put testkey 43") client2.succeed("[ $(consul kv get testkey) == 43 ]") client2.succeed("consul kv delete testkey") # Restart crashed machine. server.start() if proper_rolling_procedure: # Wait for recovery. wait_for_healthy_servers() else: # NOT proper rolling upgrade procedure, see above. wait_for_all_machines_alive() # Wait for client connections. client1.wait_until_succeeds("consul kv get -recurse") client2.wait_until_succeeds("consul kv get -recurse") # Do some consul actions with server back up. client1.succeed("consul kv put testkey 44") client2.succeed("[ $(consul kv get testkey) == 44 ]") client2.succeed("consul kv delete testkey") def all_servers_crash_simultaneously_test(): """ Tests that the cluster will eventually come back after all servers crash simultaneously. """ for server in servers: server.crash() for server in servers: server.start() </ins> # Wait for recovery. <del> for m in machines: m.wait_until_succeeds("[ $(consul members | grep -o alive | wc -l) == 5 ]") </del> <ins> wait_for_healthy_servers() </ins> # Wait for client connections. client1.wait_until_succeeds("consul kv get -recurse") client2.wait_until_succeeds("consul kv get -recurse") <del> # Do some consul actions with server back up. </del> <ins> # Do some consul actions with servers back up. </ins> client1.succeed("consul kv put testkey 44") client2.succeed("[ $(consul kv get testkey) == 44 ]") client2.succeed("consul kv delete testkey") <ins> # Run the tests. print("rolling_reboot_test()") rolling_reboot_test() print("all_servers_crash_simultaneously_test()") all_servers_crash_simultaneously_test() print("rolling_reboot_test(proper_rolling_procedure=False)") rolling_reboot_test(proper_rolling_procedure=False) </ins> ''; })

Discussion:
The I wrote is failing on (): The problem is that when machines reboot in quick succession, consensus (and cluster leadership) is lost permanently, and 2-of-3 servers cannot continue to serve requests. I have debugged it and found that it is already broken since (), but it was noticed only now. :red_circle: This is why we should run the relevant tests for every PR and should show big red warning signs when they fail! In any case, I have bisected the problem to be commit , introduced between Consul and . I have filed an upstream bug: I also found a way to make the test pass: But I don't think that's good, because I think this is a fundamental issue with Consul that affects operations (the 2-of-3 HA promise being broken), so I think it should be fundamentally fixed, and that the test is right. NixOS 20.03 still has Consul so it does not suffer from this bug, but the next NixOS release will. CC as previous consule package contributors.
IMO we shouldn't even use attr, but always refer to concrete version, like or . Seamless major consul upgrade is fine in docs, but disaster in reality (happened twice to me). That doesn't quite solve this particular problem, I know.
Yes, the key thing is that Consul doesn't permit downgrades of its state in . I would thus also always recommend to upgrade non-point-releases separately from NixOS upgrades.
Honestly I kinda wanna make a few changes to most of the Hashi packages and tests soon to provide more stable versioning (and reliable testing). I'll start cleaning it up this weekend hopefully
Upstream has a great explanation and recommended fix: Test fix and more tests in PR . Key change: . I recommend every Consul user to read that linked comment and the updated test to understand the upstream-recommended rolling reboot/upgrade procedure.