Repository: adap/flower

Issue:
If gRPC server is running behing a load balancer, clients may not be able to connect
### Describe the bug

Flower is using the `peer()` method from `grpc.ServicerContext` in order to identify unique flower clients. However, in some situations (like when running the gRPC server behind a load balancer or proxy) different clients can have the same peer identifier (i.e. the same `IP:PORT`), as HTTP/2 connections are multiplexed.

### Steps/Code to Reproduce

This can be tested with a simple [Traefik](https://doc.traefik.io/traefik/) setup as follows (https://doc.traefik.io/traefik/user-guides/grpc/):

`traefik.yaml`:
```
entryPoints:
  web:
    address: :80

providers:
  file:
    filename: grpc.yaml

api: {}
```

`grpc.yaml`:
```
http:
  routers:
    to-grpc:
      entryPoints: 
        - "web"
      service: "srv-grpc"
      rule: "HostRegexp(`.*`)"

  services:
    srv-grpc:
      loadBalancer:
        servers:
           - url: h2c://127.0.0.1:5000
```

The Traefik server can be executed as follows:
```
 ./traefik --configFile=traefik.yaml
```

Then we can launch then a Flower sever with the following code:
```
import flwr as fl

def acc_wavg(metrics):
    n = sum([i for i, _ in metrics])
    acc = sum([i * metric["accuracy"] / n for i, metric in metrics])
    return {"accuracy": acc}
 
strategy = fl.server.strategy.FedAvg(
    min_available_clients=2,
    evaluate_metrics_aggregation_fn=acc_wavg
)
 
fl.server.start_server(
    server_address="0.0.0.0:5000",
    config=fl.server.ServerConfig(num_rounds=3),
    strategy=strategy,
)
```

### Expected Results

When we connect two clients to the loadbalancer IP, on port 80, the first one connects and keeps waiting:

```
INFO flwr 2023-06-28 10:03:14,848 | grpc.py:50 | Opened insecure gRPC connection (no certificates were passed)
DEBUG flwr 2023-06-28 10:03:14,849 | connection.py:39 | ChannelConnectivity.IDLE
DEBUG flwr 2023-06-28 10:03:14,849 | connection.py:39 | ChannelConnectivity.CONNECTING
DEBUG flwr 2023-06-28 10:03:14,875 | connection.py:39 | ChannelConnectivity.READY
```

However, the second one is not connected:
```
DEBUG flwr 2023-06-28 09:37:32,554 | connection.py:39 | ChannelConnectivity.IDLE
DEBUG flwr 2023-06-28 09:37:32,554 | connection.py:39 | ChannelConnectivity.CONNECTING
DEBUG flwr 2023-06-28 09:37:32,578 | connection.py:39 | ChannelConnectivity.READY
DEBUG flwr 2023-06-28 09:37:32,784 | connection.py:113 | gRPC channel closed
Traceback (most recent call last):
  File "/home/alvaro/w/prj/fl/fl.py", line 68, in <module>
    fl.client.start_numpy_client(
  File "/home/alvaro/w/prj/fl/VENV/lib/python3.11/site-packages/flwr/client/app.py", line 252, in start_numpy_client
    start_client(
  File "/home/alvaro/w/prj/fl/VENV/lib/python3.11/site-packages/flwr/client/app.py", line 174, in start_client
    server_message = receive()
                     ^^^^^^^^^
  File "/home/alvaro/w/prj/fl/VENV/lib/python3.11/site-packages/flwr/client/grpc_client/connection.py", line 105, in <lambda>
    receive: Callable[[], ServerMessage] = lambda: next(server_message_iterator)
                                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/alvaro/w/prj/fl/VENV/lib/python3.11/site-packages/grpc/_channel.py", line 475, in __next__
    return self._next()
           ^^^^^^^^^^^^
  File "/home/alvaro/w/prj/fl/VENV/lib/python3.11/site-packages/grpc/_channel.py", line 879, in _next
    raise StopIteration()
StopIteration
```

This is because the second client connection is discarded [here]https://github.com/adap/flower/blob/main/src/py/flwr/server/client_manager.py#L149) as it is coming from the same gRPC peer (i.e. the load balancer) due to this code [here](https://github.com/adap/flower/blob/main/src/py/flwr/server/fleet/grpc_bidi/flower_service_servicer.py#L91), as due to the above I assume that the tuple `ip:port` is not unique for gRPC clients, therefore it cannot be used to identify a unique flower client (as a matter of fact, the `ip:port` are those of the load balancer, not those of the actual client).

As a test and quick hack we have moved to use uuids to identify the clients [here](https://github.com/AI4EOSC/flower/commit/b215d9f3cce1ad8806e296db4fe105a8b7f5c6c9). To my understanding, it is safe to do that way, as the peer identifier is not used elsewhere, but I do not know if it is planned to do something else with them.

### Actual Results

Two and more clients should be able to connect when running behind a load balancer.


Discussion:

Resolving patch:
diff --git a/src/py/flwr/server/fleet/grpc_bidi/flower_service_servicer.py b/src/py/flwr/server/fleet/grpc_bidi/flower_service_servicer.py
--- a/src/py/flwr/server/fleet/grpc_bidi/flower_service_servicer.py
+++ b/src/py/flwr/server/fleet/grpc_bidi/flower_service_servicer.py
@@ -18,7 +18,7 @@
 - https://github.com/grpc/grpc/blob/master/doc/statuscodes.md
 """
 
-
+import uuid
 from typing import Callable, Iterator
 
 import grpc
@@ -88,7 +88,11 @@ def Join(  # pylint: disable=invalid-name
           wrapping the actual message
         - The `Join` method is (pretty much) unaware of the protocol
         """
-        peer: str = context.peer()
+        # NOTE(aloga): this is a hack. when running flower behind a proxy, the peer can
+        # be the same for different clients (i.e. ip:port) so we use a uuid that is
+        # unique.
+        # peer: str = context.peer()
+        peer = uuid.uuid4().hex
         bridge = self.grpc_bridge_factory()
         client_proxy = self.client_proxy_factory(peer, bridge)
         is_success = register_client_proxy(self.client_manager, client_proxy, context)


Test patch: