Repository: aio-libs-abandoned/aioredis-py

Issue:
Redis.close() doesn't close connection pool created in __init__
This results in delayed cleanup via `Connection.__del__` that causes various errors. Circular references in aioredis make it harder to debug. The problem is especially clearly seen when testing with pytest-asyncio that uses a separate event loop. Just upgrading from aioredis 1.3.1 to 2.0.0 leads to the following errors in setup for random test after the test that creates aioredis client:
```
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/aioredis/connection.py", line 759, in disconnect
    self._writer.close()
  File "/usr/local/lib/python3.8/asyncio/streams.py", line 353, in close
    return self._transport.close()
  File "/usr/local/lib/python3.8/asyncio/selector_events.py", line 690, in close
    self._loop.call_soon(self._call_connection_lost, None)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 719, in call_soon
    self._check_closed()
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 508, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
```
Here the client is created with one loop and creates transport bound to it, then `__del__` is called when some other loop is running causing this transport to close. But this transport uses the loop stored when it's created, which is already closed.

With `gc.collect()` after each test it becomes much simpler:
```
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/_pytest/runner.py", line 311, in from_call
    result: Optional[TResult] = func()
  File "/usr/local/lib/python3.8/site-packages/_pytest/runner.py", line 255, in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
  File "/usr/local/lib/python3.8/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/usr/local/lib/python3.8/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/usr/local/lib/python3.8/site-packages/pluggy/manager.py", line 84, in <lambda>
    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
  File "/usr/local/lib/python3.8/site-packages/pluggy/callers.py", line 203, in _multicall
    gen.send(outcome)
  File "/usr/local/lib/python3.8/site-packages/_pytest/unraisableexception.py", line 93, in pytest_runtest_teardown
    yield from unraisable_exception_runtest_hook()
  File "/usr/local/lib/python3.8/site-packages/_pytest/unraisableexception.py", line 78, in unraisable_exception_runtest_hook
    warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))
pytest.PytestUnraisableExceptionWarning: Exception ignored in: <socket.socket fd=-1, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6>

Traceback (most recent call last):
  File "tests/fixtures.py", line 99, in event_loop
    gc.collect()
ResourceWarning: unclosed <socket.socket fd=34, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.22.0.5', 51708), raddr=('172.22.0.3', 6379)>
```

The workaround is simple:
```
         await client.close()
+        await client.connection_pool.disconnect()
```


Discussion:
Also ran into this issue.  Thanks for the workaround.

Waiting for proper fix upstream.
Please let me know if master branch has resolved this issue.
Not really. Instead of
```
RuntimeWarning: coroutine 'Connection.disconnect' was never awaited
```
we now get
```
Task was destroyed but it is pending!
task: <Task pending name='Task-2' coro=<Connection.disconnect() done, defined at /home/alex/workspace/asphalt-redis/venv310/lib64/python3.10/site-packages/aioredis/connection.py:794> wait_for=<Future pending cb=[Task.task_wakeup()]>>
```
I think this change was supposed to be in the 2.0.1 release, but that hasn't been published to the PyPI yet.
Sorry! Thanks for telling me this! 2.0.1 has now been released.

Can you provide code to reproduce? 
I'm currently trying to write a minimal reproducing example that does not involve the Asphalt framework.
Here:
```python3
import asyncio

import aioredis


async def main():
    redis = aioredis.from_url('redis://localhost:63790')
    await redis.set('key', b'value')
    await redis.close()

loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.stop()
loop.close()
```
Explicitly calling `redis.connection_pool.disconnect()` after `redis.close()` fixes the problem. Is there a reason why `redis.close()` does not do that implicitly?

Resolving patch:
diff --git a/aioredis/client.py b/aioredis/client.py
--- a/aioredis/client.py
+++ b/aioredis/client.py
@@ -858,8 +858,16 @@ def __init__(
         health_check_interval: int = 0,
         client_name: Optional[str] = None,
         username: Optional[str] = None,
+        auto_close_connection_pool: bool = True,
     ):
         kwargs: Dict[str, Any]
+        # auto_close_connection_pool only has an effect if connection_pool is
+        # None. This is a similar feature to the missing __del__ to resolve #1103,
+        # but it accounts for whether a user wants to manually close the connection
+        # pool, as a similar feature to ConnectionPool's __del__.
+        self.auto_close_connection_pool = (
+            auto_close_connection_pool if connection_pool is None else False
+        )
         if not connection_pool:
             kwargs = {
                 "db": db,
@@ -1067,11 +1075,22 @@ def __del__(self, _warnings: Any = warnings) -> None:
             context = {"client": self, "message": self._DEL_MESSAGE}
             asyncio.get_event_loop().call_exception_handler(context)
 
-    async def close(self):
+    async def close(self, close_connection_pool: Optional[bool] = None) -> None:
+        """
+        Closes Redis client connection
+
+        :param close_connection_pool: decides whether to close the connection pool used
+        by this Redis client, overriding Redis.auto_close_connection_pool. By default,
+        let Redis.auto_close_connection_pool decide whether to close the connection pool.
+        """
         conn = self.connection
         if conn:
             self.connection = None
             await self.connection_pool.release(conn)
+        if close_connection_pool or (
+            close_connection_pool is None and self.auto_close_connection_pool
+        ):
+            await self.connection_pool.disconnect()
 
     # COMMAND EXECUTION AND PROTOCOL PARSING
     async def execute_command(self, *args, **options):


Test patch:
diff --git a/tests/test_connection_pool.py b/tests/test_connection_pool.py
--- a/tests/test_connection_pool.py
+++ b/tests/test_connection_pool.py
@@ -14,6 +14,83 @@
 pytestmark = pytest.mark.asyncio
 
 
+class TestRedisAutoReleaseConnectionPool:
+    @pytest.fixture
+    async def r(self, create_redis) -> aioredis.Redis:
+        """This is necessary since r and r2 create ConnectionPools behind the scenes"""
+        r = await create_redis()
+        r.auto_close_connection_pool = True
+        yield r
+
+    @staticmethod
+    def get_total_connected_connections(pool):
+        return len(pool._available_connections) + len(pool._in_use_connections)
+
+    @staticmethod
+    async def create_two_conn(r: aioredis.Redis):
+        if not r.single_connection_client:  # Single already initialized connection
+            r.connection = await r.connection_pool.get_connection("_")
+        return await r.connection_pool.get_connection("_")
+
+    @staticmethod
+    def has_no_connected_connections(pool: aioredis.ConnectionPool):
+        return not any(
+            x.is_connected
+            for x in pool._available_connections + list(pool._in_use_connections)
+        )
+
+    async def test_auto_disconnect_redis_created_pool(self, r: aioredis.Redis):
+        new_conn = await self.create_two_conn(r)
+        assert new_conn != r.connection
+        assert self.get_total_connected_connections(r.connection_pool) == 2
+        await r.close()
+        assert self.has_no_connected_connections(r.connection_pool)
+
+    async def test_do_not_auto_disconnect_redis_created_pool(self, r2: aioredis.Redis):
+        assert r2.auto_close_connection_pool is False, (
+            "The connection pool should not be disconnected as a manually created "
+            "connection pool was passed in in conftest.py"
+        )
+        new_conn = await self.create_two_conn(r2)
+        assert self.get_total_connected_connections(r2.connection_pool) == 2
+        await r2.close()
+        assert r2.connection_pool._in_use_connections == {new_conn}
+        assert new_conn.is_connected
+        assert len(r2.connection_pool._available_connections) == 1
+        assert r2.connection_pool._available_connections[0].is_connected
+
+    async def test_auto_release_override_true_manual_created_pool(
+        self, r: aioredis.Redis
+    ):
+        assert r.auto_close_connection_pool is True, "This is from the class fixture"
+        await self.create_two_conn(r)
+        await r.close()
+        assert self.get_total_connected_connections(r.connection_pool) == 2, (
+            "The connection pool should not be disconnected as a manually created "
+            "connection pool was passed in in conftest.py"
+        )
+        assert self.has_no_connected_connections(r.connection_pool)
+
+    @pytest.mark.parametrize("auto_close_conn_pool", [True, False])
+    async def test_close_override(self, r: aioredis.Redis, auto_close_conn_pool):
+        r.auto_close_connection_pool = auto_close_conn_pool
+        await self.create_two_conn(r)
+        await r.close(close_connection_pool=True)
+        assert self.has_no_connected_connections(r.connection_pool)
+
+    @pytest.mark.parametrize("auto_close_conn_pool", [True, False])
+    async def test_negate_auto_close_client_pool(
+        self, r: aioredis.Redis, auto_close_conn_pool
+    ):
+        r.auto_close_connection_pool = auto_close_conn_pool
+        new_conn = await self.create_two_conn(r)
+        await r.close(close_connection_pool=False)
+        assert not self.has_no_connected_connections(r.connection_pool)
+        assert r.connection_pool._in_use_connections == {new_conn}
+        assert r.connection_pool._available_connections[0].is_connected
+        assert self.get_total_connected_connections(r.connection_pool) == 2
+
+
 class DummyConnection(Connection):
     description_format = "DummyConnection<>"