Repository: redis/redis-py

Issue:
RedisCluster scan_iter does not return all keys from all nodes
**Version**: What redis-py and what redis version is the issue happening on?

`redis-py==4.1.2` and `redis-cli 6.2.6`

**Platform**: What platform / version? (For example Python 3.5.1 on Windows 7 / Ubuntu 15.10 / Azure)

Python 3.7.12 on MacOS 11.4

**Description**: I started a local Redis cluster with 6 nodes (3 master nodes with ports 6001-6003, 3 slave nodes with ports 6004-6006). I wrote 19 total items to the Redis cluster: 1 is stored by 6001, 13 by 6002, and 5 by 6003. I confirmed this by doing

```
$ redis-cli -c -p 6001           
127.0.0.1:6001> KEYS *
1) "\x02\x00\x00\x00\x04\x00\x00\x00\x04\x00\x00\x00\x8b\x13\x00\x00integration_test_5d7072_1"
127.0.0.1:6001>
```
for each of the 3 master nodes.

However, when I call `client.scan_iter(b"*")`, I get a list containing only a single key, which is the key from node 6001. Here my `client` object was constructed as follows:

```
startup_nodes = [{'host': '127.0.0.1', 'port': '6001'}, {'host': '127.0.0.1', 'port': '6002'}, {'host': '127.0.0.1', 'port': '6003'}]
kwargs["startup_nodes"] = [
    ClusterNode(**node) for node in startup_nodes
]
self._client = RedisCluster(**kwargs)
```

This looks to me like there is an issue with `scan_iter` on RedisClusters, where it doesn't return all keys from all nodes.



Discussion:
Update: I did not realize that `scan_iter` would not automatically hit all nodes in the cluster. I fixed this problem by passing in `target_nodes=RedisCluster.ALL_NODES` as a kwarg. This usage is documented [here](https://github.com/redis/redis-py#cluster-mode), although I do feel like the documentation could be better (it took me a while to realize my mistake).

I have since found a related issue: `scan_iter` is not properly iterating through all keys. For example, the 6002 node has 14 keys. I can access the 6002 node by doing `node = client.get_primaries()[1]`. Then I call `scan_iter` on it by doing `client.scan(cursor="0", target_nodes=[node])`, which yields a list of length 10. This is wrong, however, as in the terminal I can confirm the node has more than 10 keys:
```
$ redis-cli -p 6002 SCAN 0
1) "3"
2)  1) "\x02\x00\x00\x00driver_id\x04\x00\x00\x00\x04\x00\x00\x00\x8b\x13\x00\x00integration_test_6011b4_1"
    2) "\x02\x00\x00\x00driver_id\x04\x00\x00\x00\x04\x00\x00\x00\x8e\x13\x00\x00integration_test_6011b4_1"
    3) "\x02\x00\x00\x00driver_id\x04\x00\x00\x00\x04\x00\x00\x00\x8f\x13\x00\x00integration_test_6011b4_1"
    4) "\x02\x00\x00\x00driver_id\x04\x00\x00\x00\x04\x00\x00\x00\x93\x13\x00\x00integration_test_6011b4_1"
    5) "\x02\x00\x00\x00driver_id\x04\x00\x00\x00\x04\x00\x00\x00\x8c\x13\x00\x00integration_test_6011b4_1"
    6) "\x02\x00\x00\x00driver_id\x04\x00\x00\x00\x04\x00\x00\x00\x99\x13\x00\x00integration_test_6011b4_1"
    7) "\x02\x00\x00\x00driver_id\x04\x00\x00\x00\x04\x00\x00\x00\x96\x13\x00\x00integration_test_6011b4_1"
    8) "\x02\x00\x00\x00driver_id\x04\x00\x00\x00\x04\x00\x00\x00\x9b\x13\x00\x00integration_test_6011b4_1"
    9) "\x02\x00\x00\x00driver_id\x04\x00\x00\x00\x04\x00\x00\x00\x9a\x13\x00\x00integration_test_6011b4_1"
   10) "\x02\x00\x00\x00driver_id\x04\x00\x00\x00\x04\x00\x00\x00\x90\x13\x00\x00integration_test_6011b4_1"
$ redis-cli -p 6002 SCAN 3
1) "0"
2) 1) "\x02\x00\x00\x00driver_id\x04\x00\x00\x00\x04\x00\x00\x00\x8a\x13\x00\x00integration_test_6011b4_1"
   2) "\x02\x00\x00\x00driver_id\x04\x00\x00\x00\x04\x00\x00\x00\x95\x13\x00\x00integration_test_6011b4_1"
   3) "\x02\x00\x00\x00driver_id\x04\x00\x00\x00\x04\x00\x00\x00\x8d\x13\x00\x00integration_test_6011b4_1"
   4) "\x02\x00\x00\x00driver_id\x04\x00\x00\x00\x04\x00\x00\x00\x89\x13\x00\x00integration_test_6011b4_1"
```

I believe the source of this issue is that `scan_iter` relies on `scan`, which itself is incorrect (at least when specifying `target_nodes`). For example, calling `result = client.scan(target_nodes=[node])` yields a tuple of the form `(0, list of 10 keys)`, which indicates that the cursor being returned is a `0`, which is not at all expected.
Just as a followup, a rollback to redis-py-cluster resolved the issues that we were facing in our integration tests which is more evidence that there may be some implementation bug in the scanning mechanism on clusters of nodes. 

This is using redis 3.5.3 and redis-py-cluster 2.1.3.
> Update: I did not realize that `scan_iter` would not automatically hit all nodes in the cluster. I fixed this problem by passing in `target_nodes=RedisCluster.ALL_NODES` as a kwarg. This usage is documented [here](https://github.com/redis/redis-py#cluster-mode), although I do feel like the documentation could be better (it took me a while to realize my mistake).

I think you're right, the documentation could definitely be better. Generally cluster support is just started! While we worked with @grokzen and @barshaul on moving this in, it's definitely not done!  Any chance @felixwang9817 you'd like to contribute a docs PR that you think might better help everyone out?

Resolving patch:
diff --git a/redis/cluster.py b/redis/cluster.py
--- a/redis/cluster.py
+++ b/redis/cluster.py
@@ -7,7 +7,7 @@
 import time
 from collections import OrderedDict
 
-from redis.client import CaseInsensitiveDict, PubSub, Redis
+from redis.client import CaseInsensitiveDict, PubSub, Redis, parse_scan
 from redis.commands import CommandsParser, RedisClusterCommands
 from redis.connection import ConnectionPool, DefaultParser, Encoder, parse_url
 from redis.crc import REDIS_CLUSTER_HASH_SLOTS, key_slot
@@ -51,10 +51,14 @@ def get_connection(redis_node, *args, **options):
 
 
 def parse_scan_result(command, res, **options):
-    keys_list = []
-    for primary_res in res.values():
-        keys_list += primary_res[1]
-    return 0, keys_list
+    cursors = {}
+    ret = []
+    for node_name, response in res.items():
+        cursor, r = parse_scan(response, **options)
+        cursors[node_name] = cursor
+        ret += r
+
+    return cursors, ret
 
 
 def parse_pubsub_numsub(command, res, **options):
@@ -244,7 +248,6 @@ class RedisCluster(RedisClusterCommands):
                 "INFO",
                 "SHUTDOWN",
                 "KEYS",
-                "SCAN",
                 "DBSIZE",
                 "BGSAVE",
                 "SLOWLOG GET",
@@ -298,6 +301,7 @@ class RedisCluster(RedisClusterCommands):
                 "FUNCTION LIST",
                 "FUNCTION LOAD",
                 "FUNCTION RESTORE",
+                "SCAN",
                 "SCRIPT EXISTS",
                 "SCRIPT FLUSH",
                 "SCRIPT LOAD",
diff --git a/redis/commands/cluster.py b/redis/commands/cluster.py
--- a/redis/commands/cluster.py
+++ b/redis/commands/cluster.py
@@ -1,5 +1,8 @@
+from typing import Iterator, Union
+
 from redis.crc import key_slot
 from redis.exceptions import RedisClusterException, RedisError
+from redis.typing import PatternT
 
 from .core import (
     ACLCommands,
@@ -206,6 +209,41 @@ def stralgo(
             **kwargs,
         )
 
+    def scan_iter(
+        self,
+        match: Union[PatternT, None] = None,
+        count: Union[int, None] = None,
+        _type: Union[str, None] = None,
+        **kwargs,
+    ) -> Iterator:
+        # Do the first query with cursor=0 for all nodes
+        cursors, data = self.scan(match=match, count=count, _type=_type, **kwargs)
+        yield from data
+
+        cursors = {name: cursor for name, cursor in cursors.items() if cursor != 0}
+        if cursors:
+            # Get nodes by name
+            nodes = {name: self.get_node(node_name=name) for name in cursors.keys()}
+
+            # Iterate over each node till its cursor is 0
+            kwargs.pop("target_nodes", None)
+            while cursors:
+                for name, cursor in cursors.items():
+                    cur, data = self.scan(
+                        cursor=cursor,
+                        match=match,
+                        count=count,
+                        _type=_type,
+                        target_nodes=nodes[name],
+                        **kwargs,
+                    )
+                    yield from data
+                    cursors[name] = cur[name]
+
+                cursors = {
+                    name: cursor for name, cursor in cursors.items() if cursor != 0
+                }
+
 
 class RedisClusterCommands(
     ClusterMultiKeyCommands,


Test patch:
diff --git a/tests/test_cluster.py b/tests/test_cluster.py
--- a/tests/test_cluster.py
+++ b/tests/test_cluster.py
@@ -1773,29 +1773,60 @@ def test_cluster_scan(self, r):
         r.set("a", 1)
         r.set("b", 2)
         r.set("c", 3)
-        cursor, keys = r.scan(target_nodes="primaries")
-        assert cursor == 0
-        assert set(keys) == {b"a", b"b", b"c"}
-        _, keys = r.scan(match="a", target_nodes="primaries")
-        assert set(keys) == {b"a"}
+
+        for target_nodes, nodes in zip(
+            ["primaries", "replicas"], [r.get_primaries(), r.get_replicas()]
+        ):
+            cursors, keys = r.scan(target_nodes=target_nodes)
+            assert sorted(keys) == [b"a", b"b", b"c"]
+            assert sorted(cursors.keys()) == sorted(node.name for node in nodes)
+            assert all(cursor == 0 for cursor in cursors.values())
+
+            cursors, keys = r.scan(match="a*", target_nodes=target_nodes)
+            assert sorted(keys) == [b"a"]
+            assert sorted(cursors.keys()) == sorted(node.name for node in nodes)
+            assert all(cursor == 0 for cursor in cursors.values())
 
     @skip_if_server_version_lt("6.0.0")
     def test_cluster_scan_type(self, r):
         r.sadd("a-set", 1)
+        r.sadd("b-set", 1)
+        r.sadd("c-set", 1)
         r.hset("a-hash", "foo", 2)
         r.lpush("a-list", "aux", 3)
-        _, keys = r.scan(match="a*", _type="SET", target_nodes="primaries")
-        assert set(keys) == {b"a-set"}
+
+        for target_nodes, nodes in zip(
+            ["primaries", "replicas"], [r.get_primaries(), r.get_replicas()]
+        ):
+            cursors, keys = r.scan(_type="SET", target_nodes=target_nodes)
+            assert sorted(keys) == [b"a-set", b"b-set", b"c-set"]
+            assert sorted(cursors.keys()) == sorted(node.name for node in nodes)
+            assert all(cursor == 0 for cursor in cursors.values())
+
+            cursors, keys = r.scan(_type="SET", match="a*", target_nodes=target_nodes)
+            assert sorted(keys) == [b"a-set"]
+            assert sorted(cursors.keys()) == sorted(node.name for node in nodes)
+            assert all(cursor == 0 for cursor in cursors.values())
 
     @skip_if_server_version_lt("2.8.0")
     def test_cluster_scan_iter(self, r):
-        r.set("a", 1)
-        r.set("b", 2)
-        r.set("c", 3)
-        keys = list(r.scan_iter(target_nodes="primaries"))
-        assert set(keys) == {b"a", b"b", b"c"}
-        keys = list(r.scan_iter(match="a", target_nodes="primaries"))
-        assert set(keys) == {b"a"}
+        keys_all = []
+        keys_1 = []
+        for i in range(100):
+            s = str(i)
+            r.set(s, 1)
+            keys_all.append(s.encode("utf-8"))
+            if s.startswith("1"):
+                keys_1.append(s.encode("utf-8"))
+        keys_all.sort()
+        keys_1.sort()
+
+        for target_nodes in ["primaries", "replicas"]:
+            keys = r.scan_iter(target_nodes=target_nodes)
+            assert sorted(keys) == keys_all
+
+            keys = r.scan_iter(match="1*", target_nodes=target_nodes)
+            assert sorted(keys) == keys_1
 
     def test_cluster_randomkey(self, r):
         node = r.get_node_from_key("{foo}")