Repository: litestar-org/litestar

Issue:
Bug: declaring a RedisStore in the app triggers a 500 error in the 2nd set of parameters of a parametrized test
### Description

If you have a parametrized test that uses a function-scoped fixture that uses itself a session-scoped fixture and you declare a RedisStore in the app, the 2nd test fails with `RuntimeError: Event loop is closed`

The 2nd test passes in those cases:
1. all fixtures are session-based (just comment the client fixture)
2. there is no RedisStore in the app (or replace with a MemoryStore)





### URL to code causing the issue

_No response_

### MCVE

```python
import msgspec
import pytest

from litestar import Litestar, get
from litestar.middleware.session.server_side import ServerSideSessionConfig
from litestar.stores.redis import RedisStore
from litestar.testing import AsyncTestClient


@get()
async def hget() -> int:
    return 1


class AppSettings(msgspec.Struct):
    debug: bool
    redis_url: str = "redis://localhost:6379"


def get_app(app_settings: AppSettings) -> Litestar:
    # setting up stores
    session_store = RedisStore.with_client(url=app_settings.redis_url)
    app = Litestar(route_handlers=[hget],
                   middleware=[
                       ServerSideSessionConfig().middleware,
                   ],
                   stores={"sessions": session_store, },  # comment this and 2nd test pass
                   debug=app_settings.debug
                   )
    return app


@pytest.fixture(scope="session")
def anyio_backend() -> str:
    return "asyncio"


@pytest.fixture(scope="session")
def app_settings_test():
    return AppSettings(debug=True)


@pytest.fixture(scope="session")
def app_test(app_settings_test: AppSettings):
    app = get_app(app_settings_test)
    yield app


@pytest.fixture  # add scope="session" and 2nd test pass
async def client(app_test: Litestar):
    async with AsyncTestClient(app=app_test) as c:
        yield c


@pytest.mark.anyio
@pytest.mark.parametrize("p1, p2",
                         [(1, 2), (3, 4)])
async def test_param(client: AsyncTestClient, p1: int, p2: int):
    response = await client.get("/")
    assert response.status_code == 200
```


### Steps to reproduce

```bash
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error
```


### Screenshots

```bash
"![SCREENSHOT_DESCRIPTION](SCREENSHOT_LINK.png)"
```


### Logs

_No response_

### Litestar Version

2.5.5

### Platform

- [X] Linux
- [ ] Mac
- [ ] Windows
- [ ] Other (Please specify in the description above)

<!-- POLAR PLEDGE BADGE START -->
---
> [!NOTE]  
> While we are open for sponsoring on [GitHub Sponsors](https://github.com/sponsors/litestar-org/) and 
> [OpenCollective](https://opencollective.com/litestar), we also utilize [Polar.sh](https://polar.sh/) to engage in pledge-based sponsorship.
>
> Check out all issues funded or available for funding [on our Polar.sh dashboard](https://polar.sh/litestar-org)
> * If you would like to see an issue prioritized, make a pledge towards it!
> * We receive the pledge once the issue is completed & verified
> * This, along with engagement in the community, helps us know which features are a priority to our users.

<a href="https://polar.sh/litestar-org/litestar/issues/3083">
<picture>
  <source media="(prefers-color-scheme: dark)" srcset="https://polar.sh/api/github/litestar-org/litestar/issues/3083/pledge.svg?darkmode=1">
  <img alt="Fund with Polar" src="https://polar.sh/api/github/litestar-org/litestar/issues/3083/pledge.svg">
</picture>
</a>
<!-- POLAR PLEDGE BADGE END -->



Discussion:
thinking out loud after I read [this](https://redis.readthedocs.io/en/stable/examples/asyncio_examples.html#Connecting-and-Disconnecting), shouldn't there be some close/aclose methods for stores, or at least one for the RedisStore ?
this can be circumvented in the get_app factory replacing the `RedisStore` with a `MemoryStore` only for tests but it feels like a hack
> thinking out loud after I read [this](https://redis.readthedocs.io/en/stable/examples/asyncio_examples.html#Connecting-and-Disconnecting), shouldn't there be some close/aclose methods for stores, or at least one for the RedisStore ?

I agree that we should add this to the stores. Then Litestar could close the stores on app shutdown (make it configurable as to whether the store should be closed or not).
> > thinking out loud after I read [this](https://redis.readthedocs.io/en/stable/examples/asyncio_examples.html#Connecting-and-Disconnecting), shouldn't there be some close/aclose methods for stores, or at least one for the RedisStore ?
> 
> I agree that we should add this to the stores. Then Litestar could close the stores on app shutdown (make it configurable as to whether the store should be closed or not).

I greped all python Redis store in github and nobody seems to have methods to close stores weirdly enough, so we're not alone :)
I'm not sure this is related to the issue at hand though, but from experience tests and unclosed ressources usually dont make good friends.
> thinking out loud after I read [this](https://redis.readthedocs.io/en/stable/examples/asyncio_examples.html#Connecting-and-Disconnecting), shouldn't there be some close/aclose methods for stores, or at least one for the RedisStore ?

The reason this does not exist is that the store can receive an "externally managed" `Redis` instance. The idea here was that whoever passes it in is responsible for managing its lifetime as well. While I still think this is the right thing to do, I can see how it could be an issue when the `with_client` classmethod is used, since then it's the store that's responsible for creating the client instance. 

Maybe the solution here is to add a flag to the `RedisStore` that indicates whether we should handle the client's lifetime? 

> I'm not sure this is related to the issue at hand though, but from experience tests and unclosed ressources usually dont make good friends.

There are/were quite a few other issues surrounding the async Redis client and lifetimes, e.g. it trying to perform some action in `__del__` that relied on an active async context, so I wouldn't entirely dismiss the possibility that this is also just the redis client being weird / buggy.

You could try though to manage its lifetime explicitly and see if that fixes the issue.
> You could try though to manage its lifetime explicitly and see if that fixes the issue.

something like this does work indeed, I'm not too sure this is not overly complicated though it does the job, and my app factory needs to be async now, and I need to wrap it in an async context manager that creates / closes the redis client :

```python
from contextlib import asynccontextmanager
from typing import Iterator

import msgspec
import pytest
from redis.asyncio import Redis, ConnectionPool

from litestar import Litestar, get
from litestar.middleware.session.server_side import ServerSideSessionConfig
from litestar.testing import AsyncTestClient


@get()
async def hget() -> int:
    return 1


class AppSettings(msgspec.Struct):
    debug: bool
    redis_url: str = "redis://localhost:6379"


@asynccontextmanager
async def make_session_store(app_settings: AppSettings) -> Iterator[None]:
    pool = ConnectionPool.from_url(app_settings.redis_url)
    client = Redis.from_pool(pool)
    try:
        yield
    finally:
        await client.aclose()


async def get_app(app_settings: AppSettings) -> Litestar:
    # setting up stores
    # session_store = RedisStore.with_client(url=app_settings.redis_url)
    # session_store = RedisStore(redis=Redis(), namespace="sessions")
    # session_store = MemoryStore()
    session_config = ServerSideSessionConfig()
    async with make_session_store(app_settings) as session_store:
        app = Litestar(route_handlers=[hget],
                       middleware=[
                           session_config.middleware,
                       ],
                       stores={"sessions": session_store},  # comment this and 2nd test pass
                       debug=app_settings.debug,
                       # lifespan=[partial(lifespan, app_settings=app_settings)]
                       )
        return app


@pytest.fixture(scope="session")
def anyio_backend() -> str:
    return "asyncio"


@pytest.fixture(scope="session")
def app_settings_test():
    return AppSettings(debug=True)


@pytest.fixture(scope="session")
async def app_test(app_settings_test: AppSettings):
    app = await get_app(app_settings_test)
    yield app


@pytest.fixture  # add scope="session" and 2nd test pass
async def client(app_test: Litestar):
    async with AsyncTestClient(app=app_test) as c:
        yield c


@pytest.mark.anyio(scope="session")
@pytest.mark.parametrize("p1, p2",
                         [(1, 2), (3, 4)])
async def test_param(client: AsyncTestClient, p1: int, p2: int):
    response = await client.get("/")
    assert response.status_code == 200
```
actually maybe if there is a way to lazily load the store, or use a lifespan-created redis client into the store that would be maybe cleaner, not sure...
> actually maybe if there is a way to lazily load the store, or use a lifespan-created redis client into the store that would be maybe cleaner, not sure...

I think you should be able to create the store in a sync function and then have a lifespan context manager on the app instance that takes care of its shutdown.
> > actually maybe if there is a way to lazily load the store, or use a lifespan-created redis client into the store that would be maybe cleaner, not sure...
> 
> I think you should be able to create the store in a sync function and then have a lifespan context manager on the app instance that takes care of its shutdown.

alright yeah this is much better this way indeed, I just have to access the private redis attribute from the store in the lifespan in order to close it but that solves everything and looks not ugly like the above :)

thanks for the patience )

> While I still think this is the right thing to do, I can see how it could be an issue when the with_client classmethod is used, since then it's the store that's responsible for creating the client instance.

Do you think I should maybe try "solve" this adding a note in the docs ? This is not trivial and as you said maybe unexpected at first. lmk


```
from contextlib import asynccontextmanager
from functools import partial
from typing import Iterator

import msgspec
import pytest
from redis.asyncio import Redis

from litestar import Litestar, get
from litestar.middleware.session.server_side import ServerSideSessionConfig
from litestar.stores.redis import RedisStore
from litestar.testing import AsyncTestClient


@get()
async def hget() -> int:
    return 1


class AppSettings(msgspec.Struct):
    debug: bool
    redis_url: str = "redis://localhost:6379"


@asynccontextmanager
async def lifespan(app: Litestar) -> Iterator[None]:
    try:
        yield
    finally:
        print("lifespan cleanup")
        await app.stores.get("sessions")._redis.aclose()


def get_app(app_settings: AppSettings) -> Litestar:
    # setting up stores
    # session_store = RedisStore.with_client(url=app_settings.redis_url)
    session_store = RedisStore(redis=Redis(), namespace="sessions")
    # session_store = MemoryStore()
    session_config = ServerSideSessionConfig()
    app = Litestar(route_handlers=[hget],
                   middleware=[
                       session_config.middleware,
                   ],
                   stores={"sessions": session_store},  # comment this and 2nd test pass
                   debug=app_settings.debug,
                   lifespan=[partial(lifespan, app_settings=app_settings)]
                   )
    return app


@pytest.fixture(scope="session")
def anyio_backend() -> str:
    return "asyncio"


@pytest.fixture(scope="session")
def app_settings_test():
    return AppSettings(debug=True)


@pytest.fixture(scope="session")
def app_test(app_settings_test: AppSettings):
    app = get_app(app_settings_test)
    yield app


@pytest.fixture  # add scope="session" and 2nd test pass
async def client(app_test: Litestar):
    async with AsyncTestClient(app=app_test) as c:
        yield c


@pytest.mark.anyio(scope="session")
@pytest.mark.parametrize("p1, p2",
                         [(1, 2), (3, 4)])
async def test_param(client: AsyncTestClient, p1: int, p2: int):
    response = await client.get("/")
    assert response.status_code == 200

```
> Do you think I should maybe try "solve" this adding a note in the docs ? This is not trivial and as you said maybe unexpected at first. lmk

I think there are two things that need to be done:

1. Add a note to the docs about passing in a `Redis` instance and that you're expected to handle its lifetime
2. Properly handle the lifetime of internally created `Redis` instances. Adding `lifespan` or `on_startup`/`on_shutdown` methods to the stores seems to be the most sensible option here, with a flag for the Redis store to know if we should handle its shutdown.
sounds good to me, I'm keen on tackling both, 1st one will be simpler and faster obviously
I've assigned you here. I'll leave it up to you if you want to address both in a single PR or two separate ones, both are fine with me.

Resolving patch:
diff --git a/litestar/app.py b/litestar/app.py
--- a/litestar/app.py
+++ b/litestar/app.py
@@ -388,7 +388,12 @@ def __init__(
 
         self._openapi_schema: OpenAPI | None = None
         self._debug: bool = True
+        self.stores: StoreRegistry = (
+            config.stores if isinstance(config.stores, StoreRegistry) else StoreRegistry(config.stores)
+        )
         self._lifespan_managers = config.lifespan
+        for store in self.stores._stores.values():
+            self._lifespan_managers.append(store)
         self._server_lifespan_managers = [p.server_lifespan for p in config.plugins or [] if isinstance(p, CLIPlugin)]
         self.experimental_features = frozenset(config.experimental_features or [])
         self.get_logger: GetLogger = get_logger_placeholder
@@ -471,10 +476,6 @@ def __init__(
 
         self.asgi_handler = self._create_asgi_handler()
 
-        self.stores: StoreRegistry = (
-            config.stores if isinstance(config.stores, StoreRegistry) else StoreRegistry(config.stores)
-        )
-
     @property
     @deprecated(version="2.6.0", kind="property", info="Use create_static_files router instead")
     def static_files_config(self) -> list[StaticFilesConfig]:
diff --git a/litestar/stores/base.py b/litestar/stores/base.py
--- a/litestar/stores/base.py
+++ b/litestar/stores/base.py
@@ -9,6 +9,8 @@
 from msgspec.msgpack import encode as msgpack_encode
 
 if TYPE_CHECKING:
+    from types import TracebackType
+
     from typing_extensions import Self
 
 
@@ -76,6 +78,17 @@ async def expires_in(self, key: str) -> int | None:
         """
         raise NotImplementedError
 
+    async def __aenter__(self) -> None:  # noqa: B027
+        pass
+
+    async def __aexit__(  # noqa: B027
+        self,
+        exc_type: type[BaseException] | None,
+        exc_val: BaseException | None,
+        exc_tb: TracebackType | None,
+    ) -> None:
+        pass
+
 
 class NamespacedStore(Store):
     """A subclass of :class:`Store`, offering hierarchical namespacing.
diff --git a/litestar/stores/redis.py b/litestar/stores/redis.py
--- a/litestar/stores/redis.py
+++ b/litestar/stores/redis.py
@@ -1,7 +1,7 @@
 from __future__ import annotations
 
 from datetime import timedelta
-from typing import cast
+from typing import TYPE_CHECKING, cast
 
 from redis.asyncio import Redis
 from redis.asyncio.connection import ConnectionPool
@@ -14,13 +14,18 @@
 
 __all__ = ("RedisStore",)
 
+if TYPE_CHECKING:
+    from types import TracebackType
+
 
 class RedisStore(NamespacedStore):
     """Redis based, thread and process safe asynchronous key/value store."""
 
     __slots__ = ("_redis",)
 
-    def __init__(self, redis: Redis, namespace: str | None | EmptyType = Empty) -> None:
+    def __init__(
+        self, redis: Redis, namespace: str | None | EmptyType = Empty, handle_client_shutdown: bool = False
+    ) -> None:
         """Initialize :class:`RedisStore`
 
         Args:
@@ -28,9 +33,11 @@ def __init__(self, redis: Redis, namespace: str | None | EmptyType = Empty) -> N
             namespace: A key prefix to simulate a namespace in redis. If not given,
                 defaults to ``LITESTAR``. Namespacing can be explicitly disabled by passing
                 ``None``. This will make :meth:`.delete_all` unavailable.
+            handle_client_shutdown: If ``True``, handle the shutdown of the `redis` instance automatically during the store's lifespan. Should be set to `True` unless the shutdown is handled externally
         """
         self._redis = redis
         self.namespace: str | None = value_or_default(namespace, "LITESTAR")
+        self.handle_client_shutdown = handle_client_shutdown
 
         # script to get and renew a key in one atomic step
         self._get_and_renew_script = self._redis.register_script(
@@ -64,6 +71,18 @@ def __init__(self, redis: Redis, namespace: str | None | EmptyType = Empty) -> N
         """
         )
 
+    async def _shutdown(self) -> None:
+        if self.handle_client_shutdown:
+            await self._redis.aclose(close_connection_pool=True)  # type: ignore[attr-defined]
+
+    async def __aexit__(
+        self,
+        exc_type: type[BaseException] | None,
+        exc_val: BaseException | None,
+        exc_tb: TracebackType | None,
+    ) -> None:
+        await self._shutdown()
+
     @classmethod
     def with_client(
         cls,
@@ -93,14 +112,22 @@ def with_client(
             username=username,
             password=password,
         )
-        return cls(redis=Redis(connection_pool=pool), namespace=namespace)
+        return cls(
+            redis=Redis(connection_pool=pool),
+            namespace=namespace,
+            handle_client_shutdown=True,
+        )
 
     def with_namespace(self, namespace: str) -> RedisStore:
         """Return a new :class:`RedisStore` with a nested virtual key namespace.
         The current instances namespace will serve as a prefix for the namespace, so it
         can be considered the parent namespace.
         """
-        return type(self)(redis=self._redis, namespace=f"{self.namespace}_{namespace}" if self.namespace else namespace)
+        return type(self)(
+            redis=self._redis,
+            namespace=f"{self.namespace}_{namespace}" if self.namespace else namespace,
+            handle_client_shutdown=self.handle_client_shutdown,
+        )
 
     def _make_key(self, key: str) -> str:
         prefix = f"{self.namespace}:" if self.namespace else ""


Test patch:
diff --git a/tests/unit/test_stores.py b/tests/unit/test_stores.py
--- a/tests/unit/test_stores.py
+++ b/tests/unit/test_stores.py
@@ -366,3 +366,16 @@ async def test_file_store_handle_rename_fail(file_store: FileStore, mocker: Mock
     await file_store.set("foo", "bar")
     mock_unlink.assert_called_once()
     assert Path(mock_unlink.call_args_list[0].args[0]).with_suffix("") == file_store.path.joinpath("foo")
+
+
+async def test_redis_store_with_client_shutdown() -> None:
+    redis_store = RedisStore.with_client(url="redis://localhost:6397")
+    assert await redis_store._redis.ping()
+    # remove the private shutdown and the assert below fails
+    # the check on connection is a mimic of https://github.com/redis/redis-py/blob/d529c2ad8d2cf4dcfb41bfd93ea68cfefd81aa66/tests/test_asyncio/test_connection_pool.py#L35-L39
+    await redis_store._shutdown()
+    assert not any(
+        x.is_connected
+        for x in redis_store._redis.connection_pool._available_connections
+        + list(redis_store._redis.connection_pool._in_use_connections)
+    )