Repository: locustio/locust

Issue:
Quitting master doesnt prevent its heartbeat_worker greenlet from trying to re-establish connections
```
[2022-01-14 13:20:02,660] perftest04/INFO/locust.main: Cleaning up runner...
[2022-01-14 13:20:02,651] master-machine/INFO/locust.runners: Client 'perftest04_1acb085b6d9a4d22b6ba31f0cedb5f0d' quit. Currently 0 clients connected.
[2022-01-14 13:20:02,717] perftest04/INFO/locust.main: Shutting down (exit code 0), bye.
[2022-01-14 13:20:02,717] perftest04/INFO/locust.main: Cleaning up runner...
[2022-01-14 13:20:02,710] master-machine/INFO/locust.runners: Client 'perftest04_808d502f37ec43a4978766d95fe5ae5e' quit. Currently 0 clients connected.
[2022-01-14 13:20:02,811] master-machine/ERROR/locust.runners: RPCError found when receiving from client: ZMQ network broken
[2022-01-14 13:20:02,812] master-machine/INFO/locust.main: Shutting down (exit code 0), bye.
[2022-01-14 13:20:02,812] master-machine/INFO/locust.main: Cleaning up runner...
[2022-01-14 13:20:02,812] master-machine/INFO/locust.util.exception_handler: Exception found on retry 1: -- retry after 1s
[2022-01-14 13:20:02,812] master-machine/ERROR/locust.util.exception_handler: ZMQ sent failure
Traceback (most recent call last):
  File "/Users/myusername/git/locust/locust/rpc/zmqrpc.py", line 27, in send_to_client
    self.socket.send_multipart([msg.node_id.encode(), msg.serialize()])
  File "/usr/local/lib/python3.9/site-packages/zmq/green/core.py", line 275, in send_multipart
    msg = super(_Socket, self).send_multipart(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/zmq/sugar/socket.py", line 589, in send_multipart
    self.send(msg, SNDMORE | flags, copy=copy, track=track)
  File "/usr/local/lib/python3.9/site-packages/zmq/green/core.py", line 228, in send
    msg = super(_Socket, self).send(data, flags, copy, track)
  File "/usr/local/lib/python3.9/site-packages/zmq/sugar/socket.py", line 541, in send
    return super(Socket, self).send(data, flags=flags, copy=copy, track=track)
  File "zmq/backend/cython/socket.pyx", line 718, in zmq.backend.cython.socket.Socket.send
  File "zmq/backend/cython/socket.pyx", line 759, in zmq.backend.cython.socket.Socket.send
  File "zmq/backend/cython/socket.pyx", line 135, in zmq.backend.cython.socket._check_closed
zmq.error.ZMQError: Socket operation on non-socket

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/myusername/git/locust/locust/util/exception_handler.py", line 13, in wrapper
    return function(*args, **kwargs)
  File "/Users/myusername/git/locust/locust/rpc/zmqrpc.py", line 29, in send_to_client
    raise RPCError("ZMQ sent failure") from e
locust.exception.RPCError: ZMQ sent failure
[2022-01-14 13:20:03,219] perftest03/INFO/locust.main: Shutting down (exit code 0), bye.
[2022-01-14 13:20:03,219] perftest03/INFO/locust.main: Cleaning up runner...
[2022-01-14 13:20:03,839] master-machine/INFO/locust.util.exception_handler: Exception found on retry 2: -- retry after 3s
[2022-01-14 13:20:03,839] master-machine/ERROR/locust.util.exception_handler: ZMQ sent failure
Traceback (most recent call last):
  File "/Users/myusername/git/locust/locust/rpc/zmqrpc.py", line 27, in send_to_client
    self.socket.send_multipart([msg.node_id.encode(), msg.serialize()])
  File "/usr/local/lib/python3.9/site-packages/zmq/green/core.py", line 275, in send_multipart
    msg = super(_Socket, self).send_multipart(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/zmq/sugar/socket.py", line 589, in send_multipart
    self.send(msg, SNDMORE | flags, copy=copy, track=track)
  File "/usr/local/lib/python3.9/site-packages/zmq/green/core.py", line 228, in send
    msg = super(_Socket, self).send(data, flags, copy, track)
  File "/usr/local/lib/python3.9/site-packages/zmq/sugar/socket.py", line 541, in send
    return super(Socket, self).send(data, flags=flags, copy=copy, track=track)
  File "zmq/backend/cython/socket.pyx", line 718, in zmq.backend.cython.socket.Socket.send
  File "zmq/backend/cython/socket.pyx", line 759, in zmq.backend.cython.socket.Socket.send
  File "zmq/backend/cython/socket.pyx", line 135, in zmq.backend.cython.socket._check_closed
zmq.error.ZMQError: Socket operation on non-socket

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/myusername/git/locust/locust/util/exception_handler.py", line 13, in wrapper
    return function(*args, **kwargs)
  File "/Users/myusername/git/locust/locust/rpc/zmqrpc.py", line 29, in send_to_client
    raise RPCError("ZMQ sent failure") from e
locust.exception.RPCError: ZMQ sent failure
[2022-01-14 13:20:06,841] master-machine/INFO/locust.util.exception_handler: Exception found on retry 3: -- retry after 5s
[2022-01-14 13:20:06,841] master-machine/ERROR/locust.util.exception_handler: ZMQ sent failure
Traceback (most recent call last):
  File "/Users/myusername/git/locust/locust/rpc/zmqrpc.py", line 27, in send_to_client
    self.socket.send_multipart([msg.node_id.encode(), msg.serialize()])
  File "/usr/local/lib/python3.9/site-packages/zmq/green/core.py", line 275, in send_multipart
    msg = super(_Socket, self).send_multipart(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/zmq/sugar/socket.py", line 589, in send_multipart
    self.send(msg, SNDMORE | flags, copy=copy, track=track)
  File "/usr/local/lib/python3.9/site-packages/zmq/green/core.py", line 228, in send
    msg = super(_Socket, self).send(data, flags, copy, track)
  File "/usr/local/lib/python3.9/site-packages/zmq/sugar/socket.py", line 541, in send
    return super(Socket, self).send(data, flags=flags, copy=copy, track=track)
  File "zmq/backend/cython/socket.pyx", line 718, in zmq.backend.cython.socket.Socket.send
  File "zmq/backend/cython/socket.pyx", line 759, in zmq.backend.cython.socket.Socket.send
  File "zmq/backend/cython/socket.pyx", line 135, in zmq.backend.cython.socket._check_closed
zmq.error.ZMQError: Socket operation on non-socket

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/myusername/git/locust/locust/util/exception_handler.py", line 13, in wrapper
    return function(*args, **kwargs)
  File "/Users/myusername/git/locust/locust/rpc/zmqrpc.py", line 29, in send_to_client
    raise RPCError("ZMQ sent failure") from e
locust.exception.RPCError: ZMQ sent failure
[2022-01-14 13:20:11,846] master-machine/INFO/locust.util.exception_handler: Retry failed after 3 times.
Traceback (most recent call last):
  File "/Users/myusername/git/locust/locust/rpc/zmqrpc.py", line 27, in send_to_client
    self.socket.send_multipart([msg.node_id.encode(), msg.serialize()])
  File "/usr/local/lib/python3.9/site-packages/zmq/green/core.py", line 275, in send_multipart
    msg = super(_Socket, self).send_multipart(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/zmq/sugar/socket.py", line 589, in send_multipart
    self.send(msg, SNDMORE | flags, copy=copy, track=track)
  File "/usr/local/lib/python3.9/site-packages/zmq/green/core.py", line 228, in send
    msg = super(_Socket, self).send(data, flags, copy, track)
  File "/usr/local/lib/python3.9/site-packages/zmq/sugar/socket.py", line 541, in send
    return super(Socket, self).send(data, flags=flags, copy=copy, track=track)
  File "zmq/backend/cython/socket.pyx", line 718, in zmq.backend.cython.socket.Socket.send
  File "zmq/backend/cython/socket.pyx", line 759, in zmq.backend.cython.socket.Socket.send
  File "zmq/backend/cython/socket.pyx", line 135, in zmq.backend.cython.socket._check_closed
zmq.error.ZMQError: Socket operation on non-socket

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/locust", line 33, in <module>
    sys.exit(load_entry_point('locust', 'console_scripts', 'locust')())
  File "/Users/myusername/git/locust/locust/main.py", line 483, in main
    shutdown()
  File "/Users/myusername/git/locust/locust/main.py", line 453, in shutdown
    runner.quit()
  File "/Users/myusername/git/locust/locust/runners.py", line 821, in quit
    self.server.send_to_client(Message("quit", None, client.id))
  File "/Users/myusername/git/locust/locust/util/exception_handler.py", line 13, in wrapper
    return function(*args, **kwargs)
  File "/Users/myusername/git/locust/locust/rpc/zmqrpc.py", line 29, in send_to_client
    raise RPCError("ZMQ sent failure") from e
locust.exception.RPCError: ZMQ sent failure
2022-01-14:13:20:11,969 INFO [swarm:436] Load gen master process finished (return code 1)
```
@delulu I think this was introduced in your code a long time ago, but for some reason I started seeing it just now (perhaps after switching from RHEL to Ubuntu for my workers?). I have a fix that I will PR very soon.


Discussion:

Resolving patch:
diff --git a/locust/runners.py b/locust/runners.py
--- a/locust/runners.py
+++ b/locust/runners.py
@@ -821,6 +821,8 @@ def quit(self):
             self.server.send_to_client(Message("quit", None, client.id))
         gevent.sleep(0.5)  # wait for final stats report from all workers
         self.server.close()
+        # ensure heartbeat_worker doesnt try to re-establish connection to workers and throw lots of exceptions
+        self.clients._worker_nodes = {}
         self.greenlet.kill(block=True)
 
     def check_stopped(self):
@@ -867,7 +869,13 @@ def client_listener(self):
             try:
                 client_id, msg = self.server.recv_from_client()
             except RPCError as e:
-                logger.error("RPCError found when receiving from client: %s" % (e))
+                if self.clients.ready:
+                    logger.error("RPCError found when receiving from client: %s" % (e))
+                else:
+                    logger.debug(
+                        "RPCError found when receiving from client: %s (but no clients were expected to be connected anyway)"
+                        % (e)
+                    )
                 self.connection_broken = True
                 gevent.sleep(FALLBACK_INTERVAL)
                 continue


Test patch: