Repository: ethereum/web3.py

Issue:
Unhandled exception in AsyncWeb3.persistent_websocket
* Version: 6.15.0
* Python: 3.12
* OS: osx

```
aiohttp==3.9.1
aiosignal==1.3.1
annotated-types==0.6.0
anyio==3.7.1
anytree==2.12.1
appnope==0.1.3
APScheduler==3.10.4
asttokens==2.4.1
asyncpg==0.29.0
attrs==23.2.0
bitarray==2.9.2
cachetools==5.3.2
certifi==2023.11.17
charset-normalizer==3.3.2
click==8.1.7
comm==0.2.1
cytoolz==0.12.2
dataclassy==0.11.1
debugpy==1.8.0
decorator==5.1.1
eip712==0.2.3
eth-abi==4.2.1
eth-account==0.10.0
eth-hash==0.6.0
eth-keyfile==0.7.0
eth-keys==0.5.0
eth-rlp==1.0.0
eth-typing==3.5.2
eth-utils==2.3.1
executing==2.0.1
fakeredis==2.20.1
fastapi==0.104.1
flake8==7.0.0
frozenlist==1.4.1
google-api-core==2.15.0
google-auth==2.26.2
google-cloud-appengine-logging==1.4.0
google-cloud-audit-log==0.2.5
google-cloud-core==2.4.1
google-cloud-logging==3.8.0
googleapis-common-protos==1.62.0
grpc-google-iam-v1==0.13.0
grpcio==1.60.0
grpcio-status==1.60.0
h11==0.14.0
hexbytes==0.3.1
idna==3.6
iniconfig==2.0.0
ipykernel==6.28.0
ipython==8.20.0
jedi==0.19.1
jsonschema==4.21.0
jsonschema-specifications==2023.12.1
jupyter_client==8.6.0
jupyter_core==5.7.1
lru-dict==1.2.0
matplotlib-inline==0.1.6
mccabe==0.7.0
multidict==6.0.4
nest-asyncio==1.5.9
numpy==1.26.3
packaging==23.2
pandas==2.1.4
parsimonious==0.9.0
parso==0.8.3
pexpect==4.9.0
platformdirs==4.1.0
pluggy==1.3.0
prompt-toolkit==3.0.43
proto-plus==1.23.0
protobuf==4.25.2
psutil==5.9.7
ptyprocess==0.7.0
pure-eval==0.2.2
pyasn1==0.5.1
pyasn1-modules==0.3.0
pycodestyle==2.11.1
pycryptodome==3.20.0
pydantic==2.5.3
pydantic-settings==2.0.3
pydantic_core==2.14.6
pyflakes==3.2.0
Pygments==2.17.2
pytest==7.4.4
pytest-asyncio==0.23.3
python-dateutil==2.8.2
python-dotenv==1.0.0
pytz==2023.3.post1
pyunormalize==15.1.0
PyYAML==6.0.1
pyzmq==25.1.2
redis==5.0.1
referencing==0.32.1
regex==2023.12.25
requests==2.31.0
rlp==4.0.0
rpds-py==0.17.1
rsa==4.9
six==1.16.0
sniffio==1.3.0
sortedcontainers==2.4.0
stack-data==0.6.3
starlette==0.27.0
toolz==0.12.0
tornado==6.4
traitlets==5.14.1
typing_extensions==4.9.0
tzdata==2023.4
tzlocal==5.2
urllib3==2.1.0
uvicorn==0.23.2
uvloop==0.19.0
wcwidth==0.2.13
web3==6.15.0
websockets==12.0
yarl==1.9.4
```


### What was wrong?

I'm subscribing to several hundred events via `eth_subscribe`. Immediately after subscribing, I make several hundred eth_cals from different AsyncWeb3 contracts

```python
contract_addreses = [...] # several hundred contracts
web3 = await AsyncWeb3.persistent_websocket(
        WebsocketProviderV2(f"ws://127.0.0.1:8546")
    )
await web3.eth.subscribe(
            "logs",
            {"address": contracts, "topics": [topic]},
        )
await asyncio.gather(*[web3.eth.contract(c, abi).functions.<func_name>().call() for c in contract_addreses])
```
Everything works as expected when the number of contracts/events/eth_calls is relatively small, but when it's large enough the following line raises a `TypeError` exception

https://github.com/ethereum/web3.py/blob/c496b4343c81b8aaf371cfcdddb2bcbd5c8cd4c4/web3/contract/utils.py#L111

which can be traced to 

https://github.com/ethereum/eth-abi/blob/main/eth_abi/utils/validation.py#L12

The exception: ```The `data` value must be of bytes type. Got <class 'dict'>```

`return_data` in my case was:

```python
{
  'id': 448,
  'jsonrpc': '2.0',
  'result': '0x00000000000000000000000000000000000000000000000000005e7dfa0e1cf6'
}
```
when the expected type is `HexBytes`

Either all eth_calls return the same error or all of them succeed (when the number of events/contracts/eth_calls is small)

I'm connecting to a Tenderly mainnet node.

I have not been able to replicate in a shareable script, but I'd be happy to connect privately and give you guys a walkthrough if interested.

Discussion:

Patch:
diff --git a/docs/providers.rst b/docs/providers.rst
index 8be2f907..82c3c2fd 100644
--- a/docs/providers.rst
+++ b/docs/providers.rst
@@ -236,6 +236,10 @@ Persistent Connection Providers
       connection and waiting for a response to be received from the listener task.
       Defaults to ``50.0``.
 
+    * ``request_information_cache_size`` is the size of the cache used to store
+      request information so that when a response is received, the provider knows
+      how to process it based on the original request. Defaults to ``500``.
+
     * ``subscription_response_queue_size`` is the size of the queue used to store
       subscription responses, defaults to ``500``. While messages are being consumed,
       this queue should never fill up as it is a transient queue and meant to handle
diff --git a/newsfragments/3224.feature.rst b/newsfragments/3224.feature.rst
new file mode 100644
index 00000000..2d7933d4
--- /dev/null
+++ b/newsfragments/3224.feature.rst
@@ -0,0 +1,1 @@
+Add formatters for new ``Cancun`` network upgrade block header fields: ``blobGasUsed``, ``excessBlobGas``, and ``parentBeaconBlockRoot``.
diff --git a/newsfragments/3226.feature.rst b/newsfragments/3226.feature.rst
new file mode 100644
index 00000000..45375efb
--- /dev/null
+++ b/newsfragments/3226.feature.rst
@@ -0,0 +1,1 @@
+Allow for configuring the ``request_information_cache_size`` for ``PersistentConnectionProvider`` classes. Issue a warning when the cache is full and unexpected behavior may occur.
diff --git a/web3/_utils/method_formatters.py b/web3/_utils/method_formatters.py
index b157829e..ceff1409 100644
--- a/web3/_utils/method_formatters.py
+++ b/web3/_utils/method_formatters.py
@@ -322,6 +322,9 @@ BLOCK_FORMATTERS = {
         is_not_null, apply_list_to_array_formatter(withdrawal_result_formatter)
     ),
     "withdrawalsRoot": apply_formatter_if(is_not_null, to_hexbytes(32)),
+    "blobGasUsed": to_integer_if_hex,
+    "excessBlobGas": to_integer_if_hex,
+    "parentBeaconBlockRoot": apply_formatter_if(is_not_null, to_hexbytes(32)),
 }
 
 
diff --git a/web3/providers/persistent.py b/web3/providers/persistent.py
index 1dd059f6..843ddcd7 100644
--- a/web3/providers/persistent.py
+++ b/web3/providers/persistent.py
@@ -34,11 +34,13 @@ class PersistentConnectionProvider(AsyncJSONBaseProvider, ABC):
     def __init__(
         self,
         request_timeout: float = DEFAULT_PERSISTENT_CONNECTION_TIMEOUT,
+        request_information_cache_size: int = 500,
         subscription_response_queue_size: int = 500,
     ) -> None:
         super().__init__()
         self._request_processor = RequestProcessor(
             self,
+            request_information_cache_size=request_information_cache_size,
             subscription_response_queue_size=subscription_response_queue_size,
         )
         self.request_timeout = request_timeout
diff --git a/web3/providers/websocket/request_processor.py b/web3/providers/websocket/request_processor.py
index da4ea157..5b1f4884 100644
--- a/web3/providers/websocket/request_processor.py
+++ b/web3/providers/websocket/request_processor.py
@@ -35,11 +35,14 @@ class RequestProcessor:
     def __init__(
         self,
         provider: "PersistentConnectionProvider",
+        request_information_cache_size: int = 500,
         subscription_response_queue_size: int = 500,
     ) -> None:
         self._provider = provider
 
-        self._request_information_cache: SimpleCache = SimpleCache(500)
+        self._request_information_cache: SimpleCache = SimpleCache(
+            request_information_cache_size
+        )
         self._request_response_cache: SimpleCache = SimpleCache(500)
         self._subscription_response_queue: asyncio.Queue[RPCResponse] = asyncio.Queue(
             maxsize=subscription_response_queue_size
@@ -81,6 +84,12 @@ class RequestProcessor:
             cache_key,
             request_info,
         )
+        if self._request_information_cache.is_full():
+            self._provider.logger.warning(
+                "Request information cache is full. This may result in unexpected "
+                "behavior. Consider increasing the ``request_information_cache_size`` "
+                "on the provider."
+            )
         return cache_key
 
     def _bump_cache_if_key_present(self, cache_key: str, request_id: int) -> None:
diff --git a/web3/utils/caching.py b/web3/utils/caching.py
index c315e908..c31a1e5d 100644
--- a/web3/utils/caching.py
+++ b/web3/utils/caching.py
@@ -15,6 +15,15 @@ class SimpleCache:
         self._size = size
         self._data: OrderedDict[str, Any] = OrderedDict()
 
+    def __contains__(self, key: str) -> bool:
+        return key in self._data
+
+    def __len__(self) -> int:
+        return len(self._data)
+
+    def is_full(self) -> bool:
+        return len(self._data) >= self._size
+
     def cache(self, key: str, value: Any) -> Tuple[Any, Dict[str, Any]]:
         evicted_items = None
         # If the key is already in the OrderedDict just update it
@@ -47,9 +56,3 @@ class SimpleCache:
             return None
 
         return self._data.pop(key)
-
-    def __contains__(self, key: str) -> bool:
-        return key in self._data
-
-    def __len__(self) -> int:
-        return len(self._data)


Test patch:
diff --git a/tests/core/eth-module/test_block_api.py b/tests/core/eth-module/test_block_api.py
index b9abae13..c41813ac 100644
--- a/tests/core/eth-module/test_block_api.py
+++ b/tests/core/eth-module/test_block_api.py
@@ -50,6 +50,9 @@ def test_get_block_formatters_with_null_values(w3):
         "transactions": [],
         "withdrawalsRoot": None,
         "withdrawals": [],
+        "blobGasUsed": None,
+        "excessBlobGas": None,
+        "parentBeaconBlockRoot": None,
     }
     result_middleware = construct_result_generator_middleware(
         {
@@ -115,6 +118,11 @@ def test_get_block_formatters_with_pre_formatted_values(w3):
                 "amount": "0x3f695",
             },
         ],
+        "blobGasUsed": "0x7ffff",
+        "excessBlobGas": "0x12c00000",
+        "parentBeaconBlockRoot": (
+            "0x6470e77f1b8a55a49a57b3f74c2a10a76185636d65122053752ea5e4bb4dac59"
+        ),
     }
     result_middleware = construct_result_generator_middleware(
         {
@@ -175,4 +183,9 @@ def test_get_block_formatters_with_pre_formatted_values(w3):
                 "amount": int(unformatted_values_block["withdrawals"][1]["amount"], 16),
             },
         ],
+        "blobGasUsed": int(unformatted_values_block["blobGasUsed"], 16),
+        "excessBlobGas": int(unformatted_values_block["excessBlobGas"], 16),
+        "parentBeaconBlockRoot": HexBytes(
+            unformatted_values_block["parentBeaconBlockRoot"]
+        ),
     }
diff --git a/tests/core/providers/test_wsv2_provider.py b/tests/core/providers/test_wsv2_provider.py
index a3356ebb..4052b2a1 100644
--- a/tests/core/providers/test_wsv2_provider.py
+++ b/tests/core/providers/test_wsv2_provider.py
@@ -297,3 +297,25 @@ async def test_listen_event_awaits_msg_processing_when_subscription_queue_is_ful
 
     # proper cleanup
     await async_w3.provider.disconnect()
+
+
+@pytest.mark.asyncio
+async def test_wsv2_req_info_cache_size_configuration_and_warns_when_full(caplog):
+    with patch(
+        "web3.providers.websocket.websocket_v2.connect", new=lambda *_1, **_2: _coro()
+    ):
+        async_w3 = await AsyncWeb3.persistent_websocket(
+            WebsocketProviderV2("ws://mocked", request_information_cache_size=1)
+        )
+
+    # fill the cache + check warning
+    async_w3.provider._request_processor.cache_request_information(
+        RPCEndpoint("eth_getBlockByNumber"),
+        ["latest"],
+        tuple(),
+    )
+    assert len(async_w3.provider._request_processor._request_information_cache) == 1
+    assert (
+        "Request information cache is full. This may result in unexpected behavior. "
+        "Consider increasing the ``request_information_cache_size`` on the provider."
+    ) in caplog.text