Repository: locustio/locust

Issue:
`test_start` event triggered multiple times on workers
<!-- 
If you have a general question about how to use Locust, please check Stack Overflow first https://stackoverflow.com/questions/tagged/locust

You can also ask new questions on SO, https://stackoverflow.com/questions/ask just remember to tag your question with "locust". Do not immediately post your issue here after posting to SO, wait for an answer there instead.

Use this form only for reporting actual bugs in Locust. Remember, the developers of Locust are unpaid volunteers, so make sure you have tried everything you can think of before filing a bug :) 

Always make sure you are running an up to date Locust version (pip3 install -U locust)
-->

### Describe the bug

When running locust in distributed mode, the `test_start` handler gets triggered every time new users are spawned, rather than once running once at the start of the test.

This only seems to occur if there is a `test_start` handler which takes longer than the interval between spawn events (I think 1s?).

This is problematic if the test needs to load test data from an external source and share it between the users.

To give an example of a concrete use case:
- I want the test to access load logs from an external source and replay them.
- These logs are valid for the lifetime of a test (rather than process or user) - so they can be shared between users on a worker.
- When I trigger the test the log data-store gets hammered by duplicated requests (load testing the wrong thing! :sweat_smile:).

### Expected behavior

The `test_start` event handlers should be triggered once when the test begins, allowing e.g. expensive test data loading operations to occur only once.

### Actual behavior

The `test_start` event handlers are triggered multiple times, with all but the final handler being killed prematurely.

### Steps to reproduce

Given the following locustfile:

```python
import time

from locust import HttpUser, events, runners, task


COUNT = 0

@events.test_start.add_listener
def test_start_worker(environment, **kwargs):
    if not isinstance(environment.runner, runners.WorkerRunner):
        return
    global COUNT
    COUNT += 1
    print(f"'test_start' triggered {COUNT} times.")
    time.sleep(1)  # Same result with `gevent.sleep`. Less than 1 second works fine, >=1 second causes the described behaviour.


class HelloWorldUser(HttpUser):
    @task
    def hello_world(self):
        self.client.get("/hello")
        self.client.get("/world")
```

And the following `docker-compose.yml`
```yaml
version: '3'

services:
  master:
    image: locustio/locust:2.5.1
    ports:
    - "8089:8089"
    volumes:
    - ./:/mnt/locust
    command: -f /mnt/locust/locustfile.py --master -H http://master:8089 --headless -u 10 -r 1 --run-time 1m --expect-workers 1

  worker:
    image: locustio/locust:2.5.1
    volumes:
    - ./:/mnt/locust
    command: -f /mnt/locust/locustfile.py --worker --master-host master
```

Run
```shell
docker-compose up -d
```

And the logs for the worker will display:

```
worker_1  | [2022-01-27 11:56:19,677] 26ff2afeca30/INFO/locust.main: Starting Locust 2.5.1
worker_1  | 'test_start' triggered 1 times.
worker_1  | 'test_start' triggered 2 times.
worker_1  | 'test_start' triggered 3 times.
worker_1  | 'test_start' triggered 4 times.
worker_1  | 'test_start' triggered 5 times.
worker_1  | 'test_start' triggered 6 times.
worker_1  | 'test_start' triggered 7 times.
worker_1  | 'test_start' triggered 8 times.
worker_1  | 'test_start' triggered 9 times.
worker_1  | 'test_start' triggered 10 times.
```

### Environment

- OS: Ubuntu 21.10
- Python version: 3.9.9
- Locust version: 2.5.1
- Locust command line that you ran: As described by `docker-compose.yml`
- Locust file contents (anonymized if necessary): Shown above
- docker-compose version 1.27.4
- Docker version 20.10.7


Discussion:
Interesting. Maybe something for you @mboutet ?
:memo: My current workaround for this is to spawn a background greenlet in the event handler, so that the event handler returns quickly and is not retriggered.

In the above example this would be:

```python
@events.test_start.add_listener
def test_start_worker(environment, **kwargs):
    if not isinstance(environment.runner, runners.WorkerRunner):
        return
    global COUNT
    COUNT += 1
    print(f"'test_start' triggered {COUNT} times.")
    gevent.spawn_later(0, time.sleep, 1)

```

Naturally this means any user code which depends on this completing needs to be defensive, since the user will start before this completes.

Leaving this here as it may help others!
Sorry for the delay, I'll try to take a look, but this seems strange. 
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 10 days.
This issue was closed because it has been stalled for 10 days with no activity. This does not necessarily mean that the issue is bad, but it most likely means that nobody is willing to take the time to fix it. If you have found Locust useful, then consider contributing a fix yourself!
@mboutet any chance you can have a look?
This is definitely an undesirable behaviour. However, having a blocking `test_start` hook running for 1s or more is tricky to handle with the current code:

https://github.com/locustio/locust/blob/300b93ed0ca587bda40230cb1535f470256e914f/locust/runners.py#L1097-L1113

This method is called every time a worker receives a `spawn` message from the master as I explained in https://github.com/locustio/locust/issues/1856#issuecomment-901949065. Therefore, if the `test_start` hook does not finish before the next `spawn` message, `self.worker_state = STATE_SPAWNING` never gets called.

We might be able to improve the behaviour by relocating the `self.environment.events.test_start.fire` just before the `if self.spawning_greenlet:` in:

https://github.com/locustio/locust/blob/300b93ed0ca587bda40230cb1535f470256e914f/locust/runners.py#L1194-L1200

However, a long-running `test_start` hook will still interfere with the ramp-up/down because the `spawn` messages from the master won't be processed until the hook is done running.

I'll open a draft PR with the change I'm proposing.

Resolving patch:
diff --git a/locust/runners.py b/locust/runners.py
--- a/locust/runners.py
+++ b/locust/runners.py
@@ -1102,15 +1102,6 @@ def start_worker(self, user_classes_count: Dict[str, int], **kwargs):
         """
         self.target_user_classes_count = user_classes_count
         self.target_user_count = sum(user_classes_count.values())
-        if self.worker_state != STATE_RUNNING and self.worker_state != STATE_SPAWNING:
-            self.stats.clear_all()
-            self.exceptions = {}
-            self.cpu_warning_emitted = False
-            self.worker_cpu_warning_emitted = False
-            self.environment._filter_tasks_by_tags()
-            self.environment.events.test_start.fire(environment=self.environment)
-
-        self.worker_state = STATE_SPAWNING
 
         for user_class in self.user_classes:
             if self.environment.host:
@@ -1192,6 +1183,16 @@ def worker(self):
                 }
                 vars(self.environment.parsed_options).update(custom_args_from_master)
 
+                if self.worker_state != STATE_RUNNING and self.worker_state != STATE_SPAWNING:
+                    self.stats.clear_all()
+                    self.exceptions = {}
+                    self.cpu_warning_emitted = False
+                    self.worker_cpu_warning_emitted = False
+                    self.environment._filter_tasks_by_tags()
+                    self.environment.events.test_start.fire(environment=self.environment)
+
+                self.worker_state = STATE_SPAWNING
+
                 if self.spawning_greenlet:
                     # kill existing spawning greenlet before we launch new one
                     self.spawning_greenlet.kill(block=True)


Test patch:
diff --git a/locust/test/test_runners.py b/locust/test/test_runners.py
--- a/locust/test/test_runners.py
+++ b/locust/test/test_runners.py
@@ -1856,6 +1856,51 @@ def on_test_start(*args, **kwargs):
 
             self.assertTrue(test_start_event_fired[0])
 
+    def test_long_running_test_start_is_run_to_completion_on_worker(self):
+        """Test for https://github.com/locustio/locust/issues/1986"""
+
+        class MyUser1(User):
+            wait_time = constant(0)
+
+            @task
+            def my_task(self):
+                pass
+
+        with mock.patch("locust.runners.WORKER_REPORT_INTERVAL", new=0.3):
+            master_env = Environment(user_classes=[MyUser1])
+            master = master_env.create_master_runner("*", 0)
+
+            sleep(0)
+
+            # start 1 worker runner
+            worker_env = Environment(user_classes=[MyUser1])
+            worker = worker_env.create_worker_runner("127.0.0.1", master.server.port)
+
+            test_start_exec_count = 0
+
+            @worker_env.events.test_start.add_listener
+            def on_test_start(*_, **__):
+                nonlocal test_start_exec_count
+                test_start_exec_count += 1
+                sleep(3)
+
+            # give worker time to connect
+            sleep(0.1)
+
+            gevent.spawn(master.start, 3, spawn_rate=1)
+
+            t0 = time.perf_counter()
+            while master.user_count != 3:
+                self.assertLessEqual(time.perf_counter() - t0, 5, "Expected 3 users to be spawned")
+                sleep(0.1)
+
+            master.quit()
+
+            # make sure users are killed
+            self.assertEqual(0, worker.user_count)
+
+            self.assertEqual(test_start_exec_count, 1)
+
 
 class TestMasterRunner(LocustRunnerTestCase):
     def setUp(self):