Repository: celery/celery

Issue:
Expose config option to set retry_policy on (Redis) ResultBackend
<!--
Please fill this template entirely and do not erase parts of it.
We reserve the right to close without a response
feature requests which are incomplete.
-->
# Checklist
<!--
To check an item on the list replace [ ] with [x].
-->

<!--
Please fill this template entirely and do not erase parts of it.
We reserve the right to close without a response
feature requests which are incomplete.
-->
# Checklist
<!--
To check an item on the list replace [ ] with [x].
-->

- [x] I have checked the [issues list](https://github.com/celery/celery/issues?utf8=%E2%9C%93&q=is%3Aissue+label%3A%22Issue+Type%3A+Feature+Request%22+) for similar or identical feature requests.
- [x] I have checked the [pull requests list](https://github.com/celery/celery/pulls?utf8=%E2%9C%93&q=is%3Apr+label%3A%22PR+Type%3A+Feature%22+) for existing proposed implementations of this feature.
- [x] I have checked the [commit log](https://github.com/celery/celery/commits/master) to find out if the if the same feature was already implemented in the master branch.
- [x] I have included all related issues and possible duplicate issues in this issue (If there are none, check this box anyway).


# Brief Summary
<!--
Please include a brief summary of what the feature does
and why it is needed.
-->

In debugging another problem I noticed that the default connection timeout for the Redis result backend is unconfigurable, and the timeout is hardcoded at 20s (20 x 1s retries)

This appears to be only actually an issue for the Redis backend (RPC is the only other backend that appears to do anything in `on_task_call`)

`task.apply_async()` calls  these functions

https://github.com/celery/celery/blob/d3863d909759f1fd618f2a1af1766ce54c16d39b/celery/app/base.py#L776-L780

https://github.com/celery/celery/blob/bf6139bf651b20bc04b895a5f6eb8d50320bc252/celery/backends/redis.py#L342-L344

https://github.com/celery/celery/blob/bf6139bf651b20bc04b895a5f6eb8d50320bc252/celery/backends/redis.py#L167-L177

https://github.com/celery/celery/blob/bf6139bf651b20bc04b895a5f6eb8d50320bc252/celery/backends/redis.py#L124-L133

`_ensure` is set via https://github.com/celery/celery/blob/bf6139bf651b20bc04b895a5f6eb8d50320bc252/celery/backends/redis.py#L97

to be https://github.com/celery/celery/blob/bf6139bf651b20bc04b895a5f6eb8d50320bc252/celery/backends/redis.py#L352-L358

But ensure is never passed any kwargs, nor is it set anywhere other than in the Backend supclass https://github.com/celery/celery/blob/bf6139bf651b20bc04b895a5f6eb8d50320bc252/celery/backends/base.py#L106-L111

# Design

The "most obvious" way of doing this would be to add a new top-level `result_backend_options` config key that feeds in to the constructor.
Expose config option to set retry_policy on (Redis) ResultBackend
<!--
Please fill this template entirely and do not erase parts of it.
We reserve the right to close without a response
feature requests which are incomplete.
-->
# Checklist
<!--
To check an item on the list replace [ ] with [x].
-->

<!--
Please fill this template entirely and do not erase parts of it.
We reserve the right to close without a response
feature requests which are incomplete.
-->
# Checklist
<!--
To check an item on the list replace [ ] with [x].
-->

- [x] I have checked the [issues list](https://github.com/celery/celery/issues?utf8=%E2%9C%93&q=is%3Aissue+label%3A%22Issue+Type%3A+Feature+Request%22+) for similar or identical feature requests.
- [x] I have checked the [pull requests list](https://github.com/celery/celery/pulls?utf8=%E2%9C%93&q=is%3Apr+label%3A%22PR+Type%3A+Feature%22+) for existing proposed implementations of this feature.
- [x] I have checked the [commit log](https://github.com/celery/celery/commits/master) to find out if the if the same feature was already implemented in the master branch.
- [x] I have included all related issues and possible duplicate issues in this issue (If there are none, check this box anyway).


# Brief Summary
<!--
Please include a brief summary of what the feature does
and why it is needed.
-->

In debugging another problem I noticed that the default connection timeout for the Redis result backend is unconfigurable, and the timeout is hardcoded at 20s (20 x 1s retries)

This appears to be only actually an issue for the Redis backend (RPC is the only other backend that appears to do anything in `on_task_call`)

`task.apply_async()` calls  these functions

https://github.com/celery/celery/blob/d3863d909759f1fd618f2a1af1766ce54c16d39b/celery/app/base.py#L776-L780

https://github.com/celery/celery/blob/bf6139bf651b20bc04b895a5f6eb8d50320bc252/celery/backends/redis.py#L342-L344

https://github.com/celery/celery/blob/bf6139bf651b20bc04b895a5f6eb8d50320bc252/celery/backends/redis.py#L167-L177

https://github.com/celery/celery/blob/bf6139bf651b20bc04b895a5f6eb8d50320bc252/celery/backends/redis.py#L124-L133

`_ensure` is set via https://github.com/celery/celery/blob/bf6139bf651b20bc04b895a5f6eb8d50320bc252/celery/backends/redis.py#L97

to be https://github.com/celery/celery/blob/bf6139bf651b20bc04b895a5f6eb8d50320bc252/celery/backends/redis.py#L352-L358

But ensure is never passed any kwargs, nor is it set anywhere other than in the Backend supclass https://github.com/celery/celery/blob/bf6139bf651b20bc04b895a5f6eb8d50320bc252/celery/backends/base.py#L106-L111

# Design

The "most obvious" way of doing this would be to add a new top-level `result_backend_options` config key that feeds in to the constructor.


Discussion:
I'll be lookin forward to a PR
Will find time to open one.

I'm still not that familiar with the internals of Celery/Kombu so wanted to check this all though first
I'll be lookin forward to a PR
Will find time to open one.

I'm still not that familiar with the internals of Celery/Kombu so wanted to check this all though first

Resolving patch:
diff --git a/celery/backends/redis.py b/celery/backends/redis.py
--- a/celery/backends/redis.py
+++ b/celery/backends/redis.py
@@ -328,6 +328,15 @@ def _params_from_url(self, url, defaults):
         connparams.update(query)
         return connparams
 
+    @cached_property
+    def retry_policy(self):
+        retry_policy = super().retry_policy
+        if "retry_policy" in self._transport_options:
+            retry_policy = retry_policy.copy()
+            retry_policy.update(self._transport_options['retry_policy'])
+
+        return retry_policy
+
     def on_task_call(self, producer, task_id):
         if not task_join_will_block():
             self.result_consumer.consume_from(task_id)
@@ -401,10 +410,11 @@ def apply_chord(self, header_result, body, **kwargs):
 
     @cached_property
     def _chord_zset(self):
-        transport_options = self.app.conf.get(
-            'result_backend_transport_options', {}
-        )
-        return transport_options.get('result_chord_ordered', True)
+        return self._transport_options.get('result_chord_ordered', True)
+
+    @cached_property
+    def _transport_options(self):
+        return self.app.conf.get('result_backend_transport_options', {})
 
     def on_chord_part_return(self, request, state, result,
                              propagate=None, **kwargs):
@@ -530,12 +540,8 @@ def _get_sentinel_instance(self, **params):
         connparams = params.copy()
 
         hosts = connparams.pop("hosts")
-        result_backend_transport_opts = self.app.conf.get(
-            "result_backend_transport_options", {})
-        min_other_sentinels = result_backend_transport_opts.get(
-            "min_other_sentinels", 0)
-        sentinel_kwargs = result_backend_transport_opts.get(
-            "sentinel_kwargs", {})
+        min_other_sentinels = self._transport_options.get("min_other_sentinels", 0)
+        sentinel_kwargs = self._transport_options.get("sentinel_kwargs", {})
 
         sentinel_instance = self.sentinel.Sentinel(
             [(cp['host'], cp['port']) for cp in hosts],
@@ -548,9 +554,7 @@ def _get_sentinel_instance(self, **params):
     def _get_pool(self, **params):
         sentinel_instance = self._get_sentinel_instance(**params)
 
-        result_backend_transport_opts = self.app.conf.get(
-            "result_backend_transport_options", {})
-        master_name = result_backend_transport_opts.get("master_name", None)
+        master_name = self._transport_options.get("master_name", None)
 
         return sentinel_instance.master_for(
             service_name=master_name,


Test patch:
diff --git a/t/unit/backends/test_redis.py b/t/unit/backends/test_redis.py
--- a/t/unit/backends/test_redis.py
+++ b/t/unit/backends/test_redis.py
@@ -523,6 +523,29 @@ def test_on_connection_error(self, logger):
         assert self.b.on_connection_error(10, exc, intervals, 3) == 30
         logger.error.assert_called_with(self.E_LOST, 3, 10, 'in 30.00 seconds')
 
+    @patch('celery.backends.redis.retry_over_time')
+    def test_retry_policy_conf(self, retry_over_time):
+        self.app.conf.result_backend_transport_options = dict(
+            retry_policy=dict(
+                max_retries=2,
+                interval_start=0,
+                interval_step=0.01,
+            ),
+        )
+        b = self.Backend(app=self.app)
+
+        def fn():
+            return 1
+
+        # We don't want to re-test retry_over_time, just check we called it
+        # with the expected args
+        b.ensure(fn, (),)
+
+        retry_over_time.assert_called_with(
+            fn, b.connection_errors, (), {}, ANY,
+            max_retries=2, interval_start=0, interval_step=0.01, interval_max=1
+        )
+
     def test_incr(self):
         self.b.client = Mock(name='client')
         self.b.incr('foo')