Repository: adap/flower

Issue:
If gRPC server is running behing a load balancer, clients may not be able to connect
### Describe the bug

Flower is using the `peer()` method from `grpc.ServicerContext` in order to identify unique flower clients. However, in some situations (like when running the gRPC server behind a load balancer or proxy) different clients can have the same peer identifier (i.e. the same `IP:PORT`), as HTTP/2 connections are multiplexed.

### Steps/Code to Reproduce

This can be tested with a simple [Traefik](https://doc.traefik.io/traefik/) setup as follows (https://doc.traefik.io/traefik/user-guides/grpc/):

`traefik.yaml`:
```
entryPoints:
  web:
    address: :80

providers:
  file:
    filename: grpc.yaml

api: {}
```

`grpc.yaml`:
```
http:
  routers:
    to-grpc:
      entryPoints: 
        - "web"
      service: "srv-grpc"
      rule: "HostRegexp(`.*`)"

  services:
    srv-grpc:
      loadBalancer:
        servers:
           - url: h2c://127.0.0.1:5000
```

The Traefik server can be executed as follows:
```
 ./traefik --configFile=traefik.yaml
```

Then we can launch then a Flower sever with the following code:
```
import flwr as fl

def acc_wavg(metrics):
    n = sum([i for i, _ in metrics])
    acc = sum([i * metric["accuracy"] / n for i, metric in metrics])
    return {"accuracy": acc}
 
strategy = fl.server.strategy.FedAvg(
    min_available_clients=2,
    evaluate_metrics_aggregation_fn=acc_wavg
)
 
fl.server.start_server(
    server_address="0.0.0.0:5000",
    config=fl.server.ServerConfig(num_rounds=3),
    strategy=strategy,
)
```

### Expected Results

When we connect two clients to the loadbalancer IP, on port 80, the first one connects and keeps waiting:

```
INFO flwr 2023-06-28 10:03:14,848 | grpc.py:50 | Opened insecure gRPC connection (no certificates were passed)
DEBUG flwr 2023-06-28 10:03:14,849 | connection.py:39 | ChannelConnectivity.IDLE
DEBUG flwr 2023-06-28 10:03:14,849 | connection.py:39 | ChannelConnectivity.CONNECTING
DEBUG flwr 2023-06-28 10:03:14,875 | connection.py:39 | ChannelConnectivity.READY
```

However, the second one is not connected:
```
DEBUG flwr 2023-06-28 09:37:32,554 | connection.py:39 | ChannelConnectivity.IDLE
DEBUG flwr 2023-06-28 09:37:32,554 | connection.py:39 | ChannelConnectivity.CONNECTING
DEBUG flwr 2023-06-28 09:37:32,578 | connection.py:39 | ChannelConnectivity.READY
DEBUG flwr 2023-06-28 09:37:32,784 | connection.py:113 | gRPC channel closed
Traceback (most recent call last):
  File "/home/alvaro/w/prj/fl/fl.py", line 68, in <module>
    fl.client.start_numpy_client(
  File "/home/alvaro/w/prj/fl/VENV/lib/python3.11/site-packages/flwr/client/app.py", line 252, in start_numpy_client
    start_client(
  File "/home/alvaro/w/prj/fl/VENV/lib/python3.11/site-packages/flwr/client/app.py", line 174, in start_client
    server_message = receive()
                     ^^^^^^^^^
  File "/home/alvaro/w/prj/fl/VENV/lib/python3.11/site-packages/flwr/client/grpc_client/connection.py", line 105, in <lambda>
    receive: Callable[[], ServerMessage] = lambda: next(server_message_iterator)
                                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/alvaro/w/prj/fl/VENV/lib/python3.11/site-packages/grpc/_channel.py", line 475, in __next__
    return self._next()
           ^^^^^^^^^^^^
  File "/home/alvaro/w/prj/fl/VENV/lib/python3.11/site-packages/grpc/_channel.py", line 879, in _next
    raise StopIteration()
StopIteration
```

This is because the second client connection is discarded [here]https://github.com/adap/flower/blob/main/src/py/flwr/server/client_manager.py#L149) as it is coming from the same gRPC peer (i.e. the load balancer) due to this code [here](https://github.com/adap/flower/blob/main/src/py/flwr/server/fleet/grpc_bidi/flower_service_servicer.py#L91), as due to the above I assume that the tuple `ip:port` is not unique for gRPC clients, therefore it cannot be used to identify a unique flower client (as a matter of fact, the `ip:port` are those of the load balancer, not those of the actual client).

As a test and quick hack we have moved to use uuids to identify the clients [here](https://github.com/AI4EOSC/flower/commit/b215d9f3cce1ad8806e296db4fe105a8b7f5c6c9). To my understanding, it is safe to do that way, as the peer identifier is not used elsewhere, but I do not know if it is planned to do something else with them.

### Actual Results

Two and more clients should be able to connect when running behind a load balancer.


Discussion:
As stated in the bug report, it seems that the peer identifier is not really needed (or used). I would be happy to submit a PR if this is the case, as otherwise it is impossible to properly run Flower behind some load balancers.

Resolving patch:
diff --git a/src/py/flwr/server/fleet/grpc_bidi/flower_service_servicer.py b/src/py/flwr/server/fleet/grpc_bidi/flower_service_servicer.py
--- a/src/py/flwr/server/fleet/grpc_bidi/flower_service_servicer.py
+++ b/src/py/flwr/server/fleet/grpc_bidi/flower_service_servicer.py
@@ -18,7 +18,7 @@
 - https://github.com/grpc/grpc/blob/master/doc/statuscodes.md
 """
 
-
+import uuid
 from typing import Callable, Iterator
 
 import grpc
@@ -91,9 +91,12 @@ def Join(  # pylint: disable=invalid-name
           wrapping the actual message
         - The `Join` method is (pretty much) unaware of the protocol
         """
-        peer: str = context.peer()
+        # When running Flower behind a proxy, the peer can be the same for
+        # different clients, so instead of `cid: str = context.peer()` we
+        # use a `UUID4` that is unique.
+        cid: str = uuid.uuid4().hex
         bridge = self.grpc_bridge_factory()
-        client_proxy = self.client_proxy_factory(peer, bridge)
+        client_proxy = self.client_proxy_factory(cid, bridge)
         is_success = register_client_proxy(self.client_manager, client_proxy, context)
 
         if is_success:


Test patch:
diff --git a/src/py/flwr/server/fleet/grpc_bidi/flower_service_servicer_test.py b/src/py/flwr/server/fleet/grpc_bidi/flower_service_servicer_test.py
--- a/src/py/flwr/server/fleet/grpc_bidi/flower_service_servicer_test.py
+++ b/src/py/flwr/server/fleet/grpc_bidi/flower_service_servicer_test.py
@@ -16,6 +16,7 @@
 
 
 import unittest
+import uuid
 from unittest.mock import MagicMock, call
 
 from flwr.proto.transport_pb2 import (  # pylint: disable=E0611
@@ -30,7 +31,8 @@
 
 CLIENT_MESSAGE = ClientMessage()
 SERVER_MESSAGE = ServerMessage()
-CLIENT_CID = "some_client_cid"
+
+CID: str = uuid.uuid4().hex
 
 
 class FlowerServiceServicerTestCase(unittest.TestCase):
@@ -42,7 +44,6 @@ def setUp(self) -> None:
         """Create mocks for tests."""
         # Mock for the gRPC context argument
         self.context_mock = MagicMock()
-        self.context_mock.peer.return_value = CLIENT_CID
 
         # Define client_messages to be processed by FlowerServiceServicer instance
         self.client_messages = [CLIENT_MESSAGE for _ in range(5)]
@@ -70,7 +71,7 @@ def setUp(self) -> None:
         # Create a GrpcClientProxy mock which we will use to test if correct
         # methods where called and client_messages are getting passed to it
         self.grpc_client_proxy_mock = MagicMock()
-        self.grpc_client_proxy_mock.cid = CLIENT_CID
+        self.grpc_client_proxy_mock.cid = CID
 
         self.client_proxy_factory_mock = MagicMock()
         self.client_proxy_factory_mock.return_value = self.grpc_client_proxy_mock
@@ -127,11 +128,7 @@ def test_join(self) -> None:
             num_server_messages += 1
 
         assert len(self.client_messages) == num_server_messages
-        assert self.grpc_client_proxy_mock.cid == CLIENT_CID
-
-        self.client_proxy_factory_mock.assert_called_once_with(
-            CLIENT_CID, self.grpc_bridge_mock
-        )
+        assert self.grpc_client_proxy_mock.cid == CID
 
         # Check if the client was registered with the client_manager
         self.client_manager_mock.register.assert_called_once_with(