Repository: redis/redis-py

Issue:
Connections are not released to a connection pool, when ClusterPipeline faces an exception on get_connection
**Version**: What redis-py and what redis version is the issue happening on?
redis-py 4.4.2 / redis 6.0.5

**Platform**: What platform / version? (For example Python 3.5.1 on Windows 7 / Ubuntu 15.10 / Azure)
Python 3.9 on MacOS Ventura 13.4.1

**Description**: Description of your issue, stack traces from errors and code that reproduces the issue

Recently, I've faced a situation where connections never get returned to a connection pool, when there is a surge of Redis commands. while investigating the incident, i've found out that the below scenario can cause a connection leak

1. ClusterPipeline is stacked with commands that needs to be directed to at least two nodes
    * let's say node 1 and node 2
3. ClusterPipeline obtains a connection to node 1 successfully
4. ClusterPipeline fails to obtain a connection to node 2 due to ConnectionError
5. ClusterPipeline retries the entire process again, but doesn't release an acquired connection to node 1 before doing that

below code includes a test case to cause a connection leak and a possible solution to the problem
- https://github.com/zakaf/redis-py/commit/eb7140cd86180bdd63bb860d4b3f32a369832bb5

Let me know, if you guys think this is an actual issue :) 



Discussion:

Resolving patch:
diff --git a/redis/cluster.py b/redis/cluster.py
--- a/redis/cluster.py
+++ b/redis/cluster.py
@@ -2143,6 +2143,8 @@ def _send_cluster_commands(
                     try:
                         connection = get_connection(redis_node, c.args)
                     except ConnectionError:
+                        for n in nodes.values():
+                            n.connection_pool.release(n.connection)
                         # Connection retries are being handled in the node's
                         # Retry object. Reinitialize the node -> slot table.
                         self.nodes_manager.initialize()


Test patch:
diff --git a/tests/test_cluster.py b/tests/test_cluster.py
--- a/tests/test_cluster.py
+++ b/tests/test_cluster.py
@@ -10,6 +10,7 @@
 from unittest.mock import DEFAULT, Mock, call, patch
 
 import pytest
+import redis
 from redis import Redis
 from redis._parsers import CommandsParser
 from redis.backoff import ExponentialBackoff, NoBackoff, default_backoff
@@ -3250,6 +3251,31 @@ def raise_ask_error():
             assert ask_node.redis_connection.connection.read_response.called
             assert res == ["MOCK_OK"]
 
+    def test_return_previously_acquired_connections(self, r):
+        # in order to ensure that a pipeline will make use of connections
+        #   from different nodes
+        assert r.keyslot("a") != r.keyslot("b")
+
+        orig_func = redis.cluster.get_connection
+        with patch("redis.cluster.get_connection") as get_connection:
+
+            def raise_error(target_node, *args, **kwargs):
+                if get_connection.call_count == 2:
+                    raise ConnectionError("mocked error")
+                else:
+                    return orig_func(target_node, *args, **kwargs)
+
+            get_connection.side_effect = raise_error
+
+            r.pipeline().get("a").get("b").execute()
+
+        # 4 = 2 get_connections per execution * 2 executions
+        assert get_connection.call_count == 4
+        for cluster_node in r.nodes_manager.nodes_cache.values():
+            connection_pool = cluster_node.redis_connection.connection_pool
+            num_of_conns = len(connection_pool._available_connections)
+            assert num_of_conns == connection_pool._created_connections
+
     def test_empty_stack(self, r):
         """
         If pipeline is executed with no commands it should