Repository: ethereum/web3.py (Python)

Issue:
Some Websocket endpoints don't handle binary communication
### What happened?

During debugging of issues of websocket connections between Ankr (which works) and NodeReal (which doesn't) with the [Subscriptions Example](https://web3py.readthedocs.io/en/latest/subscriptions.html#an-example), I noticed in the debug logs that payloads were sent in binary, vs a `websocket` (library)-based script I built whose payloads I could see in plaintext.

Long story short, it appears not all Websocket servers can handle both.

For web3py's part, `WebSocketProvider.socket_send()` requires request data to be in bytes, which according to the [`websockets` docs](https://websockets.readthedocs.io/en/stable/reference/sync/client.html#websockets.sync.client.ClientConnection.send) is then sent as a binary frame. When I modified this method to `.decode()` that payload to text, the example code worked as expected for all RPC platforms I provided.

### Full error output

```shell
web3.manager.RequestManager DEBUG    Sending request to open socket connection: [NodeReal],
    method: eth_subscribe,
    params: ('logs', {'topics': ['0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ea']})
[...]

web3.providers.WebSocketProvider DEBUG    Caching request info:
    request_id=0,
    cache_key=cfcd208495d565ef66e7dff9f98764da,
    request_info={'method': 'eth_subscribe', 'params': ('logs', {'topics': ['0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ea']}), 'response_formatters': (<function apply_formatter_if at 0x1021ea7a0>, <cyfunction identity at 0x10206b6b0>, <cyfunction identity at 0x10206b6b0>), 'subscription_id': None, 'middleware_response_processors': []}

web3.manager.RequestManager DEBUG    Getting response for request from open socket connection:
    request: {'id': 0, 'jsonrpc': '2.0', 'method': 'eth_subscribe', 'params': ('logs', {'topics': ['0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ea']})}

[...]
Timed out waiting for response with request id `0` after 30.0 second(s). This may be due to the provider not returning a response with the same id that was sent in the request or an exception raised during the request was caught and allowed to continue.
```

### Fill this section in if you know how this could or should be fixed

While I intend to relay this to NodeReal for potential infrastructure-based solutions, it may be prudent to allow users to force `WebSocketClient` to use text frames, as we have little control over which platforms support binary communication.

This could be as simple as keeping it binary internally as to not disrupt internals too much, and simply converting `request_data` to text during `.socket_send()` if a flag is present.

### web3 Version

7.8.0

### Python Version

3.11



Discussion:
Thanks for the issue. This seems like a very reasonable feature for added flexibility. We could bake in a configuration option, leaving the default as binary. I'd be happy to contribute here in the next day or two or review a PR that adds this configuration flag.



Patch:
diff --git a/docs/providers.rst b/docs/providers.rst
index 97f318ed0f..9968189be2 100644
--- a/docs/providers.rst
+++ b/docs/providers.rst
@@ -291,7 +291,7 @@ AsyncIPCProvider
 WebSocketProvider
 +++++++++++++++++
 
-.. py:class:: web3.providers.persistent.WebSocketProvider(endpoint_uri: str, websocket_kwargs: Dict[str, Any] = {})
+.. py:class:: web3.providers.persistent.WebSocketProvider(endpoint_uri: str, websocket_kwargs: Dict[str, Any] = {}, use_text_frames: bool = False)
 
     This provider handles interactions with an WS or WSS based JSON-RPC server.
 
@@ -299,6 +299,8 @@ WebSocketProvider
       ``'ws://localhost:8546'``.
     * ``websocket_kwargs`` this should be a dictionary of keyword arguments which
       will be passed onto the ws/wss websocket connection.
+    * ``use_text_frames`` will ensure websocket data is sent as text frames
+      for servers that do not support binary communication.
 
     This provider inherits from the
     :class:`~web3.providers.persistent.PersistentConnectionProvider` class. Refer to
diff --git a/newsfragments/3619.feature.rst b/newsfragments/3619.feature.rst
new file mode 100644
index 0000000000..7561aeec05
--- /dev/null
+++ b/newsfragments/3619.feature.rst
@@ -0,0 +1,1 @@
+Add `use_text_frames` flag for `WebSocketProvider` to work around websocket servers that don't support binary frames
diff --git a/web3/providers/persistent/websocket.py b/web3/providers/persistent/websocket.py
index 0a59804948..0bd7dd7995 100644
--- a/web3/providers/persistent/websocket.py
+++ b/web3/providers/persistent/websocket.py
@@ -63,6 +63,8 @@ def __init__(
         self,
         endpoint_uri: Optional[Union[URI, str]] = None,
         websocket_kwargs: Optional[Dict[str, Any]] = None,
+        # uses binary frames by default
+        use_text_frames: Optional[bool] = False,
         # `PersistentConnectionProvider` kwargs can be passed through
         **kwargs: Any,
     ) -> None:
@@ -71,6 +73,7 @@ def __init__(
             URI(endpoint_uri) if endpoint_uri is not None else get_default_endpoint()
         )
         super().__init__(**kwargs)
+        self.use_text_frames = use_text_frames
         self._ws: Optional[WebSocketClientProtocol] = None
 
         if not any(
@@ -118,9 +121,11 @@ async def socket_send(self, request_data: bytes) -> None:
                 "Connection to websocket has not been initiated for the provider."
             )
 
-        await asyncio.wait_for(
-            self._ws.send(request_data), timeout=self.request_timeout
-        )
+        payload: Union[bytes, str] = request_data
+        if self.use_text_frames:
+            payload = request_data.decode("utf-8")
+
+        await asyncio.wait_for(self._ws.send(payload), timeout=self.request_timeout)
 
     async def socket_recv(self) -> RPCResponse:
         raw_response = await self._ws.recv()


Test patch:
diff --git a/tests/core/providers/test_websocket_provider.py b/tests/core/providers/test_websocket_provider.py
index 962920e4e5..bf18d10dfd 100644
--- a/tests/core/providers/test_websocket_provider.py
+++ b/tests/core/providers/test_websocket_provider.py
@@ -55,6 +55,14 @@ class WSException(Exception):
     pass
 
 
+GET_BLOCK_JSON_MESSAGE = {
+    "id": 0,
+    "jsonrpc": "2.0",
+    "method": "eth_getBlockByNumber",
+    "params": ["latest", False],
+}
+
+
 def test_get_endpoint_uri_or_ipc_path_returns_endpoint_uri():
     provider = WebSocketProvider("ws://mocked")
     assert (
@@ -67,6 +75,14 @@ def test_get_endpoint_uri_or_ipc_path_returns_endpoint_uri():
 # -- async -- #
 
 
+def test_websocket_provider_default_values():
+    ws_uri = "ws://127.0.0.1:1337"
+    with patch.dict("os.environ", {"WEB3_WS_PROVIDER_URI": ws_uri}):
+        provider = WebSocketProvider()
+        assert provider.endpoint_uri == ws_uri
+        assert provider.use_text_frames is False
+
+
 @pytest.mark.asyncio
 async def test_disconnect_cleanup():
     provider = WebSocketProvider("ws://mocked")
@@ -454,3 +470,26 @@ async def test_persistent_connection_provider_empty_batch_response():
             # assert that even though there was an error, we have reset the batching
             # state
             assert not async_w3.provider._is_batching
+
+
+@pytest.mark.parametrize(
+    "use_text_frames, expected_send_arg",
+    (
+        (False, to_bytes(text=json.dumps(GET_BLOCK_JSON_MESSAGE))),
+        (True, json.dumps(GET_BLOCK_JSON_MESSAGE)),
+    ),
+)
+@pytest.mark.asyncio
+async def test_websocket_provider_use_text_frames(use_text_frames, expected_send_arg):
+    provider = WebSocketProvider("ws://mocked", use_text_frames=use_text_frames)
+    assert provider.use_text_frames is use_text_frames
+
+    # mock provider and add a mocked response to the cache
+    _mock_ws(provider)
+    provider._ws.send = AsyncMock()
+    provider._request_processor._request_response_cache.cache(
+        generate_cache_key(0), "0x1337"
+    )
+
+    await provider.make_request(RPCEndpoint("eth_getBlockByNumber"), ["latest", False])
+    provider._ws.send.assert_called_once_with(expected_send_arg)