Repository: locustio/locust

Issue:
Distributed test stopped despite workers running
### Describe the bug
When executing a distributed load test where worker node might not heartbeat back in-time (which is not configurable anymore) due to CPU and/or I/O-intensive tasks, it can happen that the whole test is being stopped despite the workers being fine and just busy.

### Expected behavior
The test continues to run.

### Actual behavior
All workers are being stopped by the master after the following messages:
```
[2021-02-18 10:58:33,241] 7c22a81c40a0/INFO/locust.runners: Worker ffaeb7471fb6_898127e830cc4c7487b6674f88b045fc failed to send heartbeat, setting state to missing.
[2021-02-18 10:58:33,241] 7c22a81c40a0/INFO/locust.runners: Worker da57de88394e_76e79054084547768aa00e0adba033bf failed to send heartbeat, setting state to missing.
[2021-02-18 10:58:33,241] 7c22a81c40a0/INFO/locust.runners: Worker d3c53c424e43_1644c39706a44f118090761360c76fe1 failed to send heartbeat, setting state to missing.
[2021-02-18 10:58:34,241] 7c22a81c40a0/INFO/locust.runners: Worker 95ac4f8adc8a_ce1bb0094a494d7f8a0540ebab54e105 failed to send heartbeat, setting state to missing.
[2021-02-18 10:58:34,242] 7c22a81c40a0/INFO/locust.runners: Worker c08fe40ccea4_b63905214f7846cea3f18cf529cb8767 failed to send heartbeat, setting state to missing.
[2021-02-18 10:58:35,242] 7c22a81c40a0/INFO/locust.runners: Worker 8784191196a2_9a136837b45d468cb46b0edaa3c3697c failed to send heartbeat, setting state to missing.
[2021-02-18 10:58:35,243] 7c22a81c40a0/INFO/locust.runners: Worker a5c9ae640c92_1f323758e42c4b609fa5a050c28bac50 failed to send heartbeat, setting state to missing.
[2021-02-18 10:58:36,243] 7c22a81c40a0/INFO/locust.runners: Worker 3f2bf4b8fc3f_90f59e8d0efb4e6fb424fb2e95c8c50a failed to send heartbeat, setting state to missing.
[2021-02-18 10:58:36,243] 7c22a81c40a0/INFO/locust.runners: The last worker went missing, stopping test.
```

After logging some more internals, it became evident that the calculation is simply wrong:
```
... The last worker went missing, stopping test (workers: 15, missing: 15).
```
...where:
* `workers = self.worker_count` (despite actually running 30 workers in my case)
* `missing = len(self.clients.missing)`

...however, `self.worker_count` doesn't even include missing clients, which makes the condition completely obsolete:
```python
if self.worker_count - len(self.clients.missing) <= 0:
```

So, either `self.worker_count` needs to include missing clients or the condition should changed to this instead:
```python
if self.worker_count <= 0:
```

### Steps to reproduce
Create a load-test that has a CPU-intensive task that runs for more than 3 seconds on each worker.

### Environment

- OS: Ubuntu 20.04 LTS
- Python version: 3.8
- Locust version: 1.4.3
- Locust command line that you ran: `docker-compose up --scale worker=30` (see docker-compose file below)
- Locust file contents (anonymized if necessary): the one that I have is too complex for this

Docker compose file:
```yaml
version: "3.7"

x-base-service: &base_service
  image: "locustio/locust:latest"
  restart: "no"
  volumes:
    - ./:/mnt/tests:ro
  working_dir: "/mnt/tests"

services:
  locust-master:
    <<: *base_service
    container_name: locust-master
    command: [
      "--master",
      "--headless",
      "--locustfile", "/mnt/tests/${LOCUST_FILE:?Locustfile not specified}",
      "--users", "${NUM_USERS:-10}",
      "--spawn-rate", "${SPAWN_RATE:-7}",
      "--run-time", "${RUN_TIME:-5m}",
      "--stop-timeout", "${STOP_TIMEOUT:-60}",
      "--expect-workers", "${LOCUST_WORKERS:-1}",
      "--host", "${LOCUST_TARGET:?No test target host specified}"
    ]

  worker:
    <<: *base_service
    command: [
      "--worker",
      "--master-host", "locust-master",
      "--locustfile", "/mnt/tests/${LOCUST_FILE:?Locustfile not specified}",
      "--users", "${NUM_USERS:-10}",
      "--spawn-rate", "${SPAWN_RATE:-7}",
      "--host", "${LOCUST_TARGET:?No test target host specified}"
    ]
```

`.env` file for the specific run:
```dotenv
LOCUST_FILE=<redacted>
LOCUST_WORKERS=30
NUM_USERS=240
SPAWN_RATE=30
RUN_TIME=15m
STOP_TIMEOUT=60
LOCUST_TARGET=<redacted>
```



Discussion:

Resolving patch:
diff --git a/locust/runners.py b/locust/runners.py
--- a/locust/runners.py
+++ b/locust/runners.py
@@ -608,7 +608,7 @@ def heartbeat_worker(self):
                     logger.info("Worker %s failed to send heartbeat, setting state to missing." % str(client.id))
                     client.state = STATE_MISSING
                     client.user_count = 0
-                    if self.worker_count - len(self.clients.missing) <= 0:
+                    if self.worker_count <= 0:
                         logger.info("The last worker went missing, stopping test.")
                         self.stop()
                         self.check_stopped()
@@ -654,7 +654,15 @@ def client_listener(self):
                 if msg.node_id in self.clients:
                     c = self.clients[msg.node_id]
                     c.heartbeat = HEARTBEAT_LIVENESS
-                    c.state = msg.data["state"]
+                    client_state = msg.data["state"]
+                    if c.state == STATE_MISSING:
+                        logger.info(
+                            "Worker %s self-healed with heartbeat, setting state to %s." % (str(c.id), client_state)
+                        )
+                        user_count = msg.data.get("count")
+                        if user_count:
+                            c.user_count = user_count
+                    c.state = client_state
                     c.cpu_usage = msg.data["current_cpu_usage"]
                     if not c.cpu_warning_emitted and c.cpu_usage > 90:
                         self.worker_cpu_warning_emitted = True  # used to fail the test in the end
@@ -751,7 +759,11 @@ def heartbeat(self):
                 self.client.send(
                     Message(
                         "heartbeat",
-                        {"state": self.worker_state, "current_cpu_usage": self.current_cpu_usage},
+                        {
+                            "state": self.worker_state,
+                            "current_cpu_usage": self.current_cpu_usage,
+                            "count": self.user_count,
+                        },
                         self.client_id,
                     )
                 )


Test patch:
diff --git a/locust/test/test_runners.py b/locust/test/test_runners.py
--- a/locust/test/test_runners.py
+++ b/locust/test/test_runners.py
@@ -22,6 +22,7 @@
     STATE_SPAWNING,
     STATE_RUNNING,
     STATE_MISSING,
+    STATE_STOPPING,
     STATE_STOPPED,
 )
 from locust.stats import RequestStats
@@ -754,20 +755,45 @@ def test_last_worker_missing_stops_test(self):
             master = self.get_runner()
             server.mocked_send(Message("client_ready", None, "fake_client1"))
             server.mocked_send(Message("client_ready", None, "fake_client2"))
+            server.mocked_send(Message("client_ready", None, "fake_client3"))
 
-            master.start(1, 2)
+            master.start(3, 3)
             server.mocked_send(Message("spawning", None, "fake_client1"))
             server.mocked_send(Message("spawning", None, "fake_client2"))
+            server.mocked_send(Message("spawning", None, "fake_client3"))
+
+            sleep(0.2)
+            server.mocked_send(
+                Message("heartbeat", {"state": STATE_RUNNING, "current_cpu_usage": 50, "count": 1}, "fake_client1")
+            )
+            server.mocked_send(
+                Message("heartbeat", {"state": STATE_RUNNING, "current_cpu_usage": 50, "count": 1}, "fake_client2")
+            )
+            server.mocked_send(
+                Message("heartbeat", {"state": STATE_RUNNING, "current_cpu_usage": 50, "count": 1}, "fake_client3")
+            )
 
-            sleep(0.3)
-            server.mocked_send(Message("heartbeat", {"state": STATE_RUNNING, "current_cpu_usage": 50}, "fake_client1"))
+            sleep(0.2)
+            self.assertEqual(0, len(master.clients.missing))
+            self.assertEqual(3, master.worker_count)
+            self.assertNotIn(
+                master.state, [STATE_STOPPED, STATE_STOPPING], "Not all workers went missing but test stopped anyway."
+            )
 
-            sleep(0.3)
-            self.assertEqual(1, len(master.clients.missing))
-            self.assertNotEqual(STATE_STOPPED, master.state, "Not all workers went missing but test stopped anyway.")
+            server.mocked_send(
+                Message("heartbeat", {"state": STATE_RUNNING, "current_cpu_usage": 50, "count": 1}, "fake_client1")
+            )
 
-            sleep(0.3)
+            sleep(0.4)
             self.assertEqual(2, len(master.clients.missing))
+            self.assertEqual(1, master.worker_count)
+            self.assertNotIn(
+                master.state, [STATE_STOPPED, STATE_STOPPING], "Not all workers went missing but test stopped anyway."
+            )
+
+            sleep(0.2)
+            self.assertEqual(3, len(master.clients.missing))
+            self.assertEqual(0, master.worker_count)
             self.assertEqual(STATE_STOPPED, master.state, "All workers went missing but test didn't stop.")
 
     def test_master_total_stats(self):