Repository: feast-dev/feast

Issue:
Materialization and serving fail when using Redis cluster for the online store
## Expected Behavior 
Using a Redis cluster as the online storage should work without errors.

## Current Behavior
Using a Redis cluster as the online storage no longer works after updating from 0.17.0 to 0.18.0. The issue happens whenever Feast tries to connect to the Redis cluster - e.g. during `feast materialize` or during feature serving. The exact error that is raised looks like this:

```
Traceback (most recent call last):
  File "/usr/local/bin/feast", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/feast/cli.py", line 444, in materialize_command
    end_date=utils.make_tzaware(datetime.fromisoformat(end_ts)),
  File "/usr/local/lib/python3.7/site-packages/feast/usage.py", line 269, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/feast/feature_store.py", line 1087, in materialize
    tqdm_builder=tqdm_builder,
  File "/usr/local/lib/python3.7/site-packages/feast/infra/passthrough_provider.py", line 168, in materialize_single_feature_view
    lambda x: pbar.update(x),
  File "/usr/local/lib/python3.7/site-packages/feast/infra/passthrough_provider.py", line 86, in online_write_batch
    self.online_store.online_write_batch(config, table, data, progress)
  File "/usr/local/lib/python3.7/site-packages/feast/usage.py", line 280, in wrapper
    raise exc.with_traceback(traceback)
  File "/usr/local/lib/python3.7/site-packages/feast/usage.py", line 269, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/feast/infra/online_stores/redis.py", line 184, in online_write_batch
    client = self._get_client(online_store_config)
  File "/usr/local/lib/python3.7/site-packages/feast/infra/online_stores/redis.py", line 164, in _get_client
    self._client = RedisCluster(**kwargs)
  File "/usr/local/lib/python3.7/site-packages/redis/cluster.py", line 517, in __init__
    **kwargs,
  File "/usr/local/lib/python3.7/site-packages/redis/cluster.py", line 1125, in __init__
    self.populate_startup_nodes(startup_nodes)
  File "/usr/local/lib/python3.7/site-packages/redis/cluster.py", line 1251, in populate_startup_nodes
    self.startup_nodes[n.name] = n
AttributeError: 'dict' object has no attribute 'name'
```

## Steps to reproduce
- Install Feast version 0.18.0
- Set up a Redis cluster as the online store:

```yaml
online_store:
  type: redis
  redis_type: redis_cluster
  connection_string: "host1:port1,host2:port2,host3:port3"
```

- Run `feast materialize`

### Specifications

- Version: 0.18.0
- Platform: Linux

## Possible Solution
Some investigation revealed that versions 0.17.0 and 0.18.0 differ in the sense that 0.18.0 uses `RedisCluster` from `redis.cluster` instead of `rediscluster`. As a result, the code [here](https://github.com/feast-dev/feast/blob/144f25c6dae6dbc4181ffa13fba1714c0baddab3/sdk/python/feast/infra/online_stores/redis.py#L164) breaks because the new `RedisCluster` instance expects to receive a list of `ClusterNode` instead of a list of dicts. Temporarily we worked around this issue by creating a custom online store (by copying all of the code in the linked file 😁 ) and changing it like so:

```diff
+ from redis.cluster import ClusterNode

...

-               kwargs["startup_nodes"] = startup_nodes
+               kwargs["startup_nodes"] = [ClusterNode(**node) for node in startup_nodes]
                self._client = RedisCluster(**kwargs)
```

This worked for us, but I'm not sure if this is the best approach in general because I'm not too familiar with the codebase yet.


Discussion:
hey @vstrimaitis thanks for reporting this issue and for the detailed investigation! we'll take a look at this ASAP

Resolving patch:
diff --git a/sdk/python/feast/infra/online_stores/redis.py b/sdk/python/feast/infra/online_stores/redis.py
--- a/sdk/python/feast/infra/online_stores/redis.py
+++ b/sdk/python/feast/infra/online_stores/redis.py
@@ -41,7 +41,7 @@
 
 try:
     from redis import Redis
-    from redis.cluster import RedisCluster
+    from redis.cluster import ClusterNode, RedisCluster
 except ImportError as e:
     from feast.errors import FeastExtrasDependencyImportError
 
@@ -160,7 +160,9 @@ def _get_client(self, online_store_config: RedisOnlineStoreConfig):
                 online_store_config.connection_string
             )
             if online_store_config.redis_type == RedisType.redis_cluster:
-                kwargs["startup_nodes"] = startup_nodes
+                kwargs["startup_nodes"] = [
+                    ClusterNode(**node) for node in startup_nodes
+                ]
                 self._client = RedisCluster(**kwargs)
             else:
                 kwargs["host"] = startup_nodes[0]["host"]


Test patch:
diff --git a/.github/workflows/pr_integration_tests.yml b/.github/workflows/pr_integration_tests.yml
--- a/.github/workflows/pr_integration_tests.yml
+++ b/.github/workflows/pr_integration_tests.yml
@@ -145,6 +145,10 @@ jobs:
         run: pip install pip-tools
       - name: Install dependencies
         run: make install-python-ci-dependencies
+      - name: Setup Redis Cluster
+        run: |
+          docker pull vishnunair/docker-redis-cluster:latest
+          docker run -d -p 6001:6379 -p 6002:6380 -p 6003:6381 -p 6004:6382 -p 6005:6383 -p 6006:6384 --name redis-cluster vishnunair/docker-redis-cluster
       - name: Test python
         if: ${{ always() }}  # this will guarantee that step won't be canceled and resources won't leak
         env:
diff --git a/sdk/python/tests/conftest.py b/sdk/python/tests/conftest.py
--- a/sdk/python/tests/conftest.py
+++ b/sdk/python/tests/conftest.py
@@ -30,6 +30,7 @@
 )
 from tests.integration.feature_repos.repo_configuration import (
     FULL_REPO_CONFIGS,
+    REDIS_CLUSTER_CONFIG,
     REDIS_CONFIG,
     Environment,
     construct_test_environment,
@@ -170,10 +171,14 @@ def cleanup():
     return e
 
 
-@pytest.fixture()
+@pytest.fixture(
+    scope="session",
+    params=[REDIS_CONFIG, REDIS_CLUSTER_CONFIG],
+    ids=[str(c) for c in [REDIS_CONFIG, REDIS_CLUSTER_CONFIG]],
+)
 def local_redis_environment(request, worker_id):
     e = construct_test_environment(
-        IntegrationTestRepoConfig(online_store=REDIS_CONFIG), worker_id=worker_id
+        IntegrationTestRepoConfig(online_store=request.param), worker_id=worker_id
     )
 
     def cleanup():
diff --git a/sdk/python/tests/integration/feature_repos/repo_configuration.py b/sdk/python/tests/integration/feature_repos/repo_configuration.py
--- a/sdk/python/tests/integration/feature_repos/repo_configuration.py
+++ b/sdk/python/tests/integration/feature_repos/repo_configuration.py
@@ -46,6 +46,12 @@
 
 DYNAMO_CONFIG = {"type": "dynamodb", "region": "us-west-2"}
 REDIS_CONFIG = {"type": "redis", "connection_string": "localhost:6379,db=0"}
+REDIS_CLUSTER_CONFIG = {
+    "type": "redis",
+    "redis_type": "redis_cluster",
+    # Redis Cluster Port Forwarding is setup in "pr_integration_tests.yaml" under "Setup Redis Cluster".
+    "connection_string": "127.0.0.1:6001,127.0.0.1:6002,127.0.0.1:6003",
+}
 
 # FULL_REPO_CONFIGS contains the repo configurations (e.g. provider, offline store,
 # online store, test data, and more parameters) that most integration tests will test
@@ -62,7 +68,9 @@
 if os.getenv("FEAST_IS_LOCAL_TEST", "False") != "True":
     DEFAULT_FULL_REPO_CONFIGS.extend(
         [
+            # Redis configurations
             IntegrationTestRepoConfig(online_store=REDIS_CONFIG),
+            IntegrationTestRepoConfig(online_store=REDIS_CLUSTER_CONFIG),
             # GCP configurations
             IntegrationTestRepoConfig(
                 provider="gcp",