Repository: ethereum/web3.py (python)

Issue:
Some Websocket endpoints don't handle binary communication
### What happened?

During debugging of issues of websocket connections between Ankr (which works) and NodeReal (which doesn't) with the [Subscriptions Example](https://web3py.readthedocs.io/en/latest/subscriptions.html#an-example), I noticed in the debug logs that payloads were sent in binary, vs a `websocket` (library)-based script I built whose payloads I could see in plaintext.

Long story short, it appears not all Websocket servers can handle both.

For web3py's part, `WebSocketProvider.socket_send()` requires request data to be in bytes, which according to the [`websockets` docs](https://websockets.readthedocs.io/en/stable/reference/sync/client.html#websockets.sync.client.ClientConnection.send) is then sent as a binary frame. When I modified this method to `.decode()` that payload to text, the example code worked as expected for all RPC platforms I provided.

### Full error output

```shell
web3.manager.RequestManager DEBUG    Sending request to open socket connection: [NodeReal],
    method: eth_subscribe,
    params: ('logs', {'topics': ['0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ea']})
[...]

web3.providers.WebSocketProvider DEBUG    Caching request info:
    request_id=0,
    cache_key=cfcd208495d565ef66e7dff9f98764da,
    request_info={'method': 'eth_subscribe', 'params': ('logs', {'topics': ['0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ea']}), 'response_formatters': (<function apply_formatter_if at 0x1021ea7a0>, <cyfunction identity at 0x10206b6b0>, <cyfunction identity at 0x10206b6b0>), 'subscription_id': None, 'middleware_response_processors': []}

web3.manager.RequestManager DEBUG    Getting response for request from open socket connection:
    request: {'id': 0, 'jsonrpc': '2.0', 'method': 'eth_subscribe', 'params': ('logs', {'topics': ['0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ea']})}

[...]
Timed out waiting for response with request id `0` after 30.0 second(s). This may be due to the provider not returning a response with the same id that was sent in the request or an exception raised during the request was caught and allowed to continue.
```

### Fill this section in if you know how this could or should be fixed

While I intend to relay this to NodeReal for potential infrastructure-based solutions, it may be prudent to allow users to force `WebSocketClient` to use text frames, as we have little control over which platforms support binary communication.

This could be as simple as keeping it binary internally as to not disrupt internals too much, and simply converting `request_data` to text during `.socket_send()` if a flag is present.

### web3 Version

7.8.0

### Python Version

3.11


Pull request:
Add flag to always convert websocket payloads to text frames
### What was wrong?

Some Websocket RPC platforms do not support binary frames, which is the internal default for communication, leading to connections that did not return data.

Closes #3619 

### How was it fixed?

Unfortunately this cannot be automatically determined without delaying the use of the websocket, so a flag was added on init that will inform the Provider to decode the binary payload into text so the underlying `websockets` library can send as such. And since all providers should handle text frames, setting this flag is not likely to have any negative impact.

### Todo:

- [X] Clean up commit history
- [X] Add or update documentation related to these changes
- [X] Add entry to the [release notes](https://github.com/ethereum/web3.py/blob/main/newsfragments/README.md)

#### Cute Animal Picture

![Put a link to a cute animal picture inside the parenthesis-->](https://media.hswstatic.com/eyJidWNrZXQiOiJjb250ZW50Lmhzd3N0YXRpYy5jb20iLCJrZXkiOiJnaWZcL2dldHR5aW1hZ2VzLTE1MDIyNDY3MzUuanBnIiwiZWRpdHMiOnsicmVzaXplIjp7IndpZHRoIjo4Mjh9LCJ0b0Zvcm1hdCI6ImF2aWYifX0=)


Patch:
diff --git a/docs/providers.rst b/docs/providers.rst
index 97f318ed..9968189b 100644
--- a/docs/providers.rst
+++ b/docs/providers.rst
@@ -291,7 +291,7 @@ AsyncIPCProvider
 WebSocketProvider
 +++++++++++++++++
 
-.. py:class:: web3.providers.persistent.WebSocketProvider(endpoint_uri: str, websocket_kwargs: Dict[str, Any] = {})
+.. py:class:: web3.providers.persistent.WebSocketProvider(endpoint_uri: str, websocket_kwargs: Dict[str, Any] = {}, use_text_frames: bool = False)
 
     This provider handles interactions with an WS or WSS based JSON-RPC server.
 
@@ -299,6 +299,8 @@ WebSocketProvider
       ``'ws://localhost:8546'``.
     * ``websocket_kwargs`` this should be a dictionary of keyword arguments which
       will be passed onto the ws/wss websocket connection.
+    * ``use_text_frames`` will ensure websocket data is sent as text frames
+      for servers that do not support binary communication.
 
     This provider inherits from the
     :class:`~web3.providers.persistent.PersistentConnectionProvider` class. Refer to
diff --git a/newsfragments/3619.feature.rst b/newsfragments/3619.feature.rst
new file mode 100644
index 00000000..7561aeec
--- /dev/null
+++ b/newsfragments/3619.feature.rst
@@ -0,0 +1,1 @@
+Add `use_text_frames` flag for `WebSocketProvider` to work around websocket servers that don't support binary frames
diff --git a/web3/providers/persistent/websocket.py b/web3/providers/persistent/websocket.py
index 0a598049..0bd7dd79 100644
--- a/web3/providers/persistent/websocket.py
+++ b/web3/providers/persistent/websocket.py
@@ -63,6 +63,8 @@ class WebSocketProvider(PersistentConnectionProvider):
         self,
         endpoint_uri: Optional[Union[URI, str]] = None,
         websocket_kwargs: Optional[Dict[str, Any]] = None,
+        # uses binary frames by default
+        use_text_frames: Optional[bool] = False,
         # `PersistentConnectionProvider` kwargs can be passed through
         **kwargs: Any,
     ) -> None:
@@ -71,6 +73,7 @@ class WebSocketProvider(PersistentConnectionProvider):
             URI(endpoint_uri) if endpoint_uri is not None else get_default_endpoint()
         )
         super().__init__(**kwargs)
+        self.use_text_frames = use_text_frames
         self._ws: Optional[WebSocketClientProtocol] = None
 
         if not any(
@@ -118,9 +121,11 @@ class WebSocketProvider(PersistentConnectionProvider):
                 "Connection to websocket has not been initiated for the provider."
             )
 
-        await asyncio.wait_for(
-            self._ws.send(request_data), timeout=self.request_timeout
-        )
+        payload: Union[bytes, str] = request_data
+        if self.use_text_frames:
+            payload = request_data.decode("utf-8")
+
+        await asyncio.wait_for(self._ws.send(payload), timeout=self.request_timeout)
 
     async def socket_recv(self) -> RPCResponse:
         raw_response = await self._ws.recv()


Test patch:
diff --git a/tests/core/providers/test_websocket_provider.py b/tests/core/providers/test_websocket_provider.py
index 962920e4..bf18d10d 100644
--- a/tests/core/providers/test_websocket_provider.py
+++ b/tests/core/providers/test_websocket_provider.py
@@ -55,6 +55,14 @@ class WSException(Exception):
     pass
 
 
+GET_BLOCK_JSON_MESSAGE = {
+    "id": 0,
+    "jsonrpc": "2.0",
+    "method": "eth_getBlockByNumber",
+    "params": ["latest", False],
+}
+
+
 def test_get_endpoint_uri_or_ipc_path_returns_endpoint_uri():
     provider = WebSocketProvider("ws://mocked")
     assert (
@@ -67,6 +75,14 @@ def test_get_endpoint_uri_or_ipc_path_returns_endpoint_uri():
 # -- async -- #
 
 
+def test_websocket_provider_default_values():
+    ws_uri = "ws://127.0.0.1:1337"
+    with patch.dict("os.environ", {"WEB3_WS_PROVIDER_URI": ws_uri}):
+        provider = WebSocketProvider()
+        assert provider.endpoint_uri == ws_uri
+        assert provider.use_text_frames is False
+
+
 @pytest.mark.asyncio
 async def test_disconnect_cleanup():
     provider = WebSocketProvider("ws://mocked")
@@ -454,3 +470,26 @@ async def test_persistent_connection_provider_empty_batch_response():
             # assert that even though there was an error, we have reset the batching
             # state
             assert not async_w3.provider._is_batching
+
+
+@pytest.mark.parametrize(
+    "use_text_frames, expected_send_arg",
+    (
+        (False, to_bytes(text=json.dumps(GET_BLOCK_JSON_MESSAGE))),
+        (True, json.dumps(GET_BLOCK_JSON_MESSAGE)),
+    ),
+)
+@pytest.mark.asyncio
+async def test_websocket_provider_use_text_frames(use_text_frames, expected_send_arg):
+    provider = WebSocketProvider("ws://mocked", use_text_frames=use_text_frames)
+    assert provider.use_text_frames is use_text_frames
+
+    # mock provider and add a mocked response to the cache
+    _mock_ws(provider)
+    provider._ws.send = AsyncMock()
+    provider._request_processor._request_response_cache.cache(
+        generate_cache_key(0), "0x1337"
+    )
+
+    await provider.make_request(RPCEndpoint("eth_getBlockByNumber"), ["latest", False])
+    provider._ws.send.assert_called_once_with(expected_send_arg)