Repository: BerriAI/litellm

Issue:
[Bug]: Redis Cache in cluster mode doesn't work in litellm router
### What happened?

Getting cached response doesn't work starting 1.61.11 version. Relevant logs are shared below.

I noticed that writing caches work without any error (And I can see respective log on our Redis), but fetching them back throws errors. I tried with 1.61.13 as well but nothing changed. Meanwhile we pinned to the 1.61.9 version which seems working.

### Relevant log output

```shell
Error occurred in async batch get cache - Event loop is closed

Error occurred in async batch get cache - CROSSSLOT Keys in request don't hash to the same slot
```

### Are you a ML Ops Team?

No

### What LiteLLM version are you on ?

1.61.11

### Twitter / LinkedIn details

_No response_

Discussion:
@ishaan-jaff: hi @ogunoz is this calling a RedisCluster ? @ishaan-jaff: should be working here: https://github.com/BerriAI/litellm/releases/tag/v1.61.13.rc @ogunoz: Hey @ishaan-jaff thank you for your answer. It is a RedisCluster yes.

I still have the error even with 1.61.15. By the way we are not using Litellm as Proxy server, we are using the Python SDK@ogunoz: Hmm even though i still get the error logs, i am also getting the response from logs actually, so it seems like less critical@ogunoz: I can verify that SET and GET calls to redis cluster works as expected that is why actual prompt caching works as expected. Issue is with MGET calls

[https://stackoverflow.com/a/79455860](https://stackoverflow.com/a/79455860)

I found few logs from Redis cluster if it gives some insight @ishaan-jaff 

```
MGET litellm:deployment:e35983666a577e9054ac068971f5c34d05ccc4ad9e1d4902df1912d360da72f5:cooldown 
litellm:deployment:85af94d8352f857079a9956535d0f4ba3dee6fbe7c6eb8a51f8f7596d89323d6:cooldown 
litellm:deployment:c5f312129dbb3c7e2d31371b0f9af3ae9756407e8d430ad671e90a1072178238:cooldown 
litellm:deployment:9bb5fe5b696f042c7f1f58608305e9dc233c25da11c6b74a28d70cb1d1bc86be:cooldown 
litellm:deployment:9bc88229eb670446f9a739bf0c4f9685a27f4e200f96598ef6e7e1ffed64bc9d:cooldown 
litellm:deployment:42ad7471ca7517557ffc48e0c9fab0d48988076274a4485dd50f4ab32e2d811e:cooldown 
litellm:deployment:0594db3bc1ee7bf62a2c3355c000099979d165eee93fd7b279962bab5a47a6f9:cooldown
```

@ogunoz: According to https://repost.aws/knowledge-center/elasticache-crossslot-keys-error-redis 

It is possible to ensure that keys are in the same MGET call will be in the same slot by setting { } characters to the common part, therefore it seems like setting that in `get_cooldown_cache_key` solves the issue. There are hardcoded cooldown cache keys in the same module, i also assume that we use the same function everywhere there (`cooldown_cache.py`)

```python
    @staticmethod
    def get_cooldown_cache_key(model_id: str) -> str:
        # Notice the { } chars
        return f"{{deployment}}:{model_id}:cooldown"
```@ishaan-jaff: For Redis cluster we don't use mget we use mget atomic @ogunoz: Ah thank you so much @ishaan-jaff for your answer. I found that once we can use `RedisClusterCache` properly, there is no such issue. However in `router.py` even though `startup_nodes` are provided in `cache_kwargs` always `RedisCache` instance created. Therefore it opts for mget instead of mget-nonatomic. Could you make a change there (`litellm.router.py` line `350`)


Even though RedisCache class instantiates all the redis clients (sync and async) as cluster instances (thanks to redis kwargs), it goes to MGET instead of mget-nonatomic.
From
```python
redis_cache = RedisCache(**cache_config)
```

To 
```python
if cache_config.get("startup_nodes"):  # or "startup_nodes" in cache_config
    redis_cache = RedisClusterCache(**cache_config)
 else:
    redis_cache = RedisCache(**cache_config)
```@ogunoz: My attempt to fix it :) https://github.com/BerriAI/litellm/pull/9010@ogunoz: Closing the issue, fixed in 1.63.3

Solution patch:
diff --git a/docs/my-website/sidebars.js b/docs/my-website/sidebars.js
index 51c4c8c21..40e8b2c6f 100644
--- a/docs/my-website/sidebars.js
+++ b/docs/my-website/sidebars.js
@@ -263,6 +263,13 @@ const sidebars = {
     {
       type: "category",
       label: "Supported Endpoints",
+      link: {
+        type: "generated-index",
+        title: "Supported Endpoints",
+        description:
+          "Learn how to deploy + call models from different providers on LiteLLM",
+        slug: "/supported_endpoints",
+      },
       items: [
         {
           type: "category",
diff --git a/litellm/router.py b/litellm/router.py
index 9ddd59600..d1c410e78 100644
--- a/litellm/router.py
+++ b/litellm/router.py
@@ -44,7 +44,12 @@ import litellm.litellm_core_utils
 import litellm.litellm_core_utils.exception_mapping_utils
 from litellm import get_secret_str
 from litellm._logging import verbose_router_logger
-from litellm.caching.caching import DualCache, InMemoryCache, RedisCache
+from litellm.caching.caching import (
+    DualCache,
+    InMemoryCache,
+    RedisCache,
+    RedisClusterCache,
+)
 from litellm.integrations.custom_logger import CustomLogger
 from litellm.litellm_core_utils.asyncify import run_async_function
 from litellm.litellm_core_utils.core_helpers import _get_parent_otel_span_from_kwargs
@@ -351,7 +356,7 @@ class Router:
 
             # Add additional key-value pairs from cache_kwargs
             cache_config.update(cache_kwargs)
-            redis_cache = RedisCache(**cache_config)
+            redis_cache = self._create_redis_cache(cache_config)
 
         if cache_responses:
             if litellm.cache is None:
@@ -611,6 +616,15 @@ class Router:
             litellm.callbacks, self
         )
 
+    @staticmethod
+    def _create_redis_cache(
+        cache_config: dict[str, Any]
+    ) -> RedisCache | RedisClusterCache:
+        if cache_config.get("startup_nodes"):
+            return RedisClusterCache(**cache_config)
+        else:
+            return RedisCache(**cache_config)
+
     def _update_redis_cache(self, cache: RedisCache):
         """
         Update the redis cache for the router, if none set.


Test patch:
diff --git a/tests/local_testing/test_router_caching.py b/tests/local_testing/test_router_caching.py
index 88e9111bf..53a79b943 100644
--- a/tests/local_testing/test_router_caching.py
+++ b/tests/local_testing/test_router_caching.py
@@ -5,6 +5,7 @@ import os
 import sys
 import time
 import traceback
+from unittest.mock import patch
 
 import pytest
 
@@ -13,6 +14,8 @@ sys.path.insert(
 )  # Adds the parent directory to the system path
 import litellm
 from litellm import Router
+from litellm.caching import RedisCache, RedisClusterCache
+
 
 ## Scenarios
 ## 1. 2 models - openai + azure - 1 model group "gpt-3.5-turbo",
@@ -322,3 +325,36 @@ async def test_acompletion_caching_on_router_caching_groups():
     except Exception as e:
         traceback.print_exc()
         pytest.fail(f"Error occurred: {e}")
+
+
+@pytest.mark.parametrize(
+    "startup_nodes, expected_cache_type",
+    [
+        pytest.param(
+            [dict(host="node1.localhost", port=6379)],
+            RedisClusterCache,
+            id="Expects a RedisClusterCache instance when startup_nodes provided",
+        ),
+        pytest.param(
+            None,
+            RedisCache,
+            id="Expects a RedisCache instance when there is no startup nodes",
+        ),
+    ],
+)
+def test_create_correct_redis_cache_instance(
+    startup_nodes: list[dict] | None,
+    expected_cache_type: type[RedisClusterCache | RedisCache],
+):
+    cache_config = dict(
+        host="mockhost",
+        port=6379,
+        password="mock-password",
+        startup_nodes=startup_nodes,
+    )
+
+    def _mock_redis_cache_init(*args, **kwargs): ...
+
+    with patch.object(RedisCache, "__init__", _mock_redis_cache_init):
+        redis_cache = Router._create_redis_cache(cache_config)
+        assert isinstance(redis_cache, expected_cache_type)