Repository: celery/celery

Issue:
Allow GroupResult.join timeout to be configurable in celery.chord_unlock
# Checklist

- [x] I have checked the issues list for similar or identical enhancement to an existing feature.
- [x] I have checked the commit log to find out if a the same enhancement was already implemented in master.

# Brief Summary

Pull request: #5348 

Previously the timeout passed down to `GroupResult.join` in `celery.chord_unlock` was hardcoded to 3.0 seconds. This introduces the new configuration option `result_chord_join_timeout` which allows users to configure the timeout. The default value remains as 3.0.

This change will solve the issue of unwanted timeouts caused when there's a moderate latency between the Celery workers and the configured backend or the result set to join is relatively large (5000+ task results to join).

# Design

## Architectural Considerations
<!--
If more components other than Celery are involved,
describe them here and the effect it would have on Celery.
-->
None

## Proposed Behavior

See summary.

## Proposed UI/UX

The introduced configuration can be applied as follows:

```python
app.conf.result_chord_join_timeout = 3.0  # default timeout
app.conf.result_chord_join_timeout = 10.0  # 10 second timeout
app.conf.result_chord_join_timeout = None  # No timeout
```

## Diagrams
<!--
Please include any diagrams that might be relevant
to the implementation of this enhancement such as:
* Class Diagrams
* Sequence Diagrams
* Activity Diagrams
You can drag and drop images into the text box to attach them to this issue.
-->
N/A

## Alternatives
<!--
If you have considered any alternative implementations
describe them in detail below.
-->
None


Discussion:

Resolving patch:
diff --git a/celery/app/builtins.py b/celery/app/builtins.py
--- a/celery/app/builtins.py
+++ b/celery/app/builtins.py
@@ -78,7 +78,10 @@ def unlock_chord(self, group_id, callback, interval=None,
         callback = maybe_signature(callback, app=app)
         try:
             with allow_join_result():
-                ret = j(timeout=3.0, propagate=True)
+                ret = j(
+                    timeout=app.conf.result_chord_join_timeout,
+                    propagate=True,
+                )
         except Exception as exc:  # pylint: disable=broad-except
             try:
                 culprit = next(deps._failed_join_report())
diff --git a/celery/app/defaults.py b/celery/app/defaults.py
--- a/celery/app/defaults.py
+++ b/celery/app/defaults.py
@@ -210,6 +210,7 @@ def __repr__(self):
         extended=Option(False, type='bool'),
         serializer=Option('json'),
         backend_transport_options=Option({}, type='dict'),
+        chord_join_timeout=Option(3.0, type='float'),
     ),
     elasticsearch=Namespace(
         __old__=old_ns('celery_elasticsearch'),


Test patch:
diff --git a/t/unit/tasks/test_chord.py b/t/unit/tasks/test_chord.py
--- a/t/unit/tasks/test_chord.py
+++ b/t/unit/tasks/test_chord.py
@@ -177,6 +177,28 @@ class NeverReady(TSR):
     def test_is_in_registry(self):
         assert 'celery.chord_unlock' in self.app.tasks
 
+    def _test_unlock_join_timeout(self, timeout):
+        class MockJoinResult(TSR):
+            is_ready = True
+            value = [(None,)]
+            join = Mock(return_value=value)
+            join_native = join
+
+        self.app.conf.result_chord_join_timeout = timeout
+        with self._chord_context(MockJoinResult):
+            MockJoinResult.join.assert_called_with(
+                timeout=timeout,
+                propagate=True,
+            )
+
+    def test_unlock_join_timeout_default(self):
+        self._test_unlock_join_timeout(
+            timeout=self.app.conf.result_chord_join_timeout,
+        )
+
+    def test_unlock_join_timeout_custom(self):
+        self._test_unlock_join_timeout(timeout=5.0)
+
 
 class test_chord(ChordCase):