Repository: ethereum/web3.py

Issue:
Async websocket subscription using 100% CPU
* Version: 7.0.0b2, 6.15.1
* Python: 3.11
* OS: linux
* `pip freeze` output

```
(web3) quenouille@LAPTOP-G5F7OIM4:~/web3.py$ pip freeze
aiohttp==3.9.3
aiosignal==1.3.1
alabaster==0.7.16
annotated-types==0.6.0
attrs==23.2.0
Babel==2.14.0
bitarray==2.9.2
black==24.3.0
blocklint==0.2.4
build==1.1.1
bump2version==1.0.1
bumpversion==0.6.0
cached-property==1.5.2
cachetools==5.3.3
certifi==2024.2.2
cffi==1.16.0
chardet==5.2.0
charset-normalizer==3.3.2
click==8.1.7
click-default-group==1.2.4
colorama==0.4.6
cryptography==42.0.5
cytoolz==0.12.3
distlib==0.3.8
docopt==0.6.2
docutils==0.20.1
eth-account==0.11.0
eth-bloom==3.0.0
eth-hash==0.7.0
eth-keyfile==0.8.0
eth-keys==0.5.0
eth-rlp==1.0.1
eth-tester==0.10.0b3
eth-typing==4.0.0
eth-utils==4.0.0
eth_abi==5.0.1
execnet==2.0.2
filelock==3.13.3
flake8==3.8.3
flaky==3.8.1
frozenlist==1.4.1
hexbytes==0.3.1
hypothesis==6.99.13
idna==3.6
imagesize==1.4.1
importlib_metadata==7.1.0
incremental==22.10.0
iniconfig==2.0.0
isort==5.13.2
jaraco.classes==3.3.1
jaraco.context==4.3.0
jaraco.functools==4.0.0
jeepney==0.8.0
Jinja2==3.1.3
keyring==25.0.0
lru-dict==1.3.0
markdown-it-py==3.0.0
MarkupSafe==2.1.5
mccabe==0.6.1
mdurl==0.1.2
more-itertools==10.2.0
multidict==6.0.5
mypy==1.4.1
mypy-extensions==1.0.0
nh3==0.2.17
packaging==24.0
parsimonious==0.9.0
pathspec==0.12.1
pkginfo==1.10.0
platformdirs==4.2.0
pluggy==1.4.0
py-ecc==7.0.0
py-evm==0.9.0b1
py-geth==4.3.0
pycodestyle==2.6.0
pycparser==2.21
pycryptodome==3.20.0
pydantic==2.6.4
pydantic_core==2.16.3
pyflakes==2.2.0
Pygments==2.17.2
pyproject-api==1.6.1
pyproject_hooks==1.0.0
pytest==8.1.1
pytest-asyncio==0.21.1
pytest-mock==3.14.0
pytest-watch==4.2.0
pytest-xdist==3.5.0
pyunormalize==15.1.0
readme_renderer==43.0
regex==2023.12.25
requests==2.31.0
requests-toolbelt==1.0.0
rfc3986==2.0.0
rich==13.7.1
rlp==4.0.0
safe-pysha3==1.0.4
SecretStorage==3.3.3
semantic-version==2.10.0
snowballstemmer==2.2.0
sortedcontainers==2.4.0
Sphinx==7.2.6
sphinx-rtd-theme==2.0.0
sphinxcontrib-applehelp==1.0.8
sphinxcontrib-devhelp==1.0.6
sphinxcontrib-htmlhelp==2.0.5
sphinxcontrib-jquery==4.1
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.7
sphinxcontrib-serializinghtml==1.1.10
tomli==2.0.1
toolz==0.12.1
towncrier==21.9.0
tox==4.14.2
tqdm==4.66.2
trie==3.0.0
twine==5.0.0
types-requests==2.31.0.20240311
types-setuptools==69.2.0.20240317
typing_extensions==4.10.0
urllib3==2.2.1
virtualenv==20.25.1
watchdog==4.0.0
-e git+ssh://git@github.com/ethereum/web3.py.git@653c536d5a2fbb31b231d39948bac1244ee622c3#egg=web3
websockets==12.0
yarl==1.9.4
zipp==3.18.1
```


### What was wrong?

Please include any of the following that are applicable:

* The code which produced the error
```
from web3 import AsyncWeb3
from web3.providers import WebSocketProvider
import asyncio

WS_PROVIDER_URL = "wss://eth-mainnet.ws.alchemyapi.io/ws/demo"

async def subscribe_to_new_block_head():
    async with AsyncWeb3(WebSocketProvider(WS_PROVIDER_URL)) as w3:
        await w3.eth.subscribe("newHeads")
        async for response in w3.socket.process_subscriptions():
            print(response)
        

if __name__ == "__main__":
    asyncio.run(subscribe_to_new_block_head())
```

* The full output of the error
Not an error, but 100% CPU usage can be observed

* What type of node you were connecting to.
Ethereum mainnet

### How can it be fixed?

---
Making RequestProcessor.pop_raw_response an async function, await pop_raw_response instead of polling it.

**Note:** We prefer to use issues to track our work. If you think you've encountered a bug in web3py or 
have a feature request, you're in the right place. If you have implementation or usage questions, 
please refer to our [documentation](https://web3py.readthedocs.io/en/latest/) and/or join the conversation 
on [discord](https://discord.gg/GHryRvPB84).



Discussion:

Resolving patch:
diff --git a/web3/manager.py b/web3/manager.py
--- a/web3/manager.py
+++ b/web3/manager.py
@@ -1,4 +1,3 @@
-import asyncio
 import logging
 from typing import (
     TYPE_CHECKING,
@@ -349,11 +348,7 @@ async def _message_stream(self) -> AsyncGenerator[RPCResponse, None]:
             )
 
         while True:
-            # sleep(0) here seems to be the most efficient way to yield control
-            # back to the event loop while waiting for the response in the queue.
-            await asyncio.sleep(0)
-
-            response = self._request_processor.pop_raw_response(subscription=True)
+            response = await self._request_processor.pop_raw_response(subscription=True)
             if (
                 response is not None
                 and response.get("params", {}).get("subscription")
diff --git a/web3/providers/persistent/persistent.py b/web3/providers/persistent/persistent.py
--- a/web3/providers/persistent/persistent.py
+++ b/web3/providers/persistent/persistent.py
@@ -78,7 +78,7 @@ async def _match_response_id_to_request_id() -> RPCResponse:
                     self.logger.debug(
                         f"Popping response for id {request_id} from cache."
                     )
-                    popped_response = self._request_processor.pop_raw_response(
+                    popped_response = await self._request_processor.pop_raw_response(
                         cache_key=request_cache_key,
                     )
                     return popped_response
diff --git a/web3/providers/persistent/request_processor.py b/web3/providers/persistent/request_processor.py
--- a/web3/providers/persistent/request_processor.py
+++ b/web3/providers/persistent/request_processor.py
@@ -235,19 +235,17 @@ async def cache_raw_response(
             )
             self._request_response_cache.cache(cache_key, raw_response)
 
-    def pop_raw_response(
+    async def pop_raw_response(
         self, cache_key: str = None, subscription: bool = False
     ) -> Any:
         if subscription:
             qsize = self._subscription_response_queue.qsize()
-            if qsize == 0:
-                return None
+            raw_response = await self._subscription_response_queue.get()
 
             if not self._provider._listen_event.is_set():
                 self._provider._listen_event.set()
 
-            raw_response = self._subscription_response_queue.get_nowait()
-            if qsize == 1:
+            if qsize == 0:
                 if not self._subscription_queue_synced_with_ws_stream:
                     self._subscription_queue_synced_with_ws_stream = True
                     self._provider.logger.info(


Test patch: