Repository: celery/celery

Issue:
revoke() does not support wait-and-retry
Potentially this is not an actual issue, but it is behavior which is not necessarily ideal, and potentially something which could be fixed.  

## Checklist

- [x] I have included the output of ``celery -A proj report`` in the issue.
      (if you are not able to do this, then at least specify the Celery
       version affected).
- [x] I have verified that the issue exists against the `master` branch of Celery.

## Steps to reproduce

```
software -> celery:3.1.20 (Cipater) kombu:3.0.33 py:2.7.13
            billiard:3.3.0.23 py-amqp:1.4.9
platform -> system:Linux arch:64bit imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:amqp results:disabled
```

Save this code as reproducer.py.  Ensure the broker is down.

```
#!/usr/env python

from celery import Celery
from celery.app import control
from celery.signals import celeryd_after_setup
from uuid import uuid4

app = Celery(broker='amqp://')


@celeryd_after_setup.connect
def delete_worker(*args, **kwargs):
    """
    Imagine if we wanted to do some cleanup, supposing that maybe things
    had previously been left in a dirty state for some reason.
    """
    controller = control.Control(app=app)
    controller.revoke(uuid4(), terminate=True)

#@worker_shutdown.connect
#def delete_worker(*args, **kwargs):
#    controller = control.Control(app=app)
#    controller.revoke(uuid4(), terminate=True)
```

We're attempting to do a bit of cleanup just in case a previous task that may have not been cleaned up properly, and to do so we're calling Control.revoke(...., terminate=True).

## Expected behavior

The celeryd_after_setup signal  supposedly takes place after the queue setup has been handled.  Therefore, I would expect that if it cannot communicate with the broker during this period it would attempt to wait and retry.

Whether the worker_shutdown case should attempt to retry is more questionable, but I still wouldn't necessarily expect it to immediately fail and traceback.

My question is: is this intentional behavior, and if it is intentional behavior, would the Celery devs consider adding retry functionality to either of these cases to be an improvement?

## Actual behavior

**celeryd_after_setup** signal:

```
[2017-06-21 22:42:52,145: ERROR/MainProcess] Unrecoverable error: error(error(111, 'Connection refused'),)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/celery/worker/__init__.py", line 206, in start
    self.blueprint.start(self)
  File "/usr/lib/python2.7/site-packages/celery/bootsteps.py", line 119, in start
    self.on_start()
  File "/usr/lib/python2.7/site-packages/celery/apps/worker.py", line 158, in on_start
    sender=self.hostname, instance=self, conf=self.app.conf,
  File "/usr/lib/python2.7/site-packages/celery/utils/dispatch/signal.py", line 166, in send
    response = receiver(signal=self, sender=sender, **named)
  File "/home/vagrant/devel/reproducer.py", line 14, in delete_worker
    controller.revoke(uuid4(), terminate=True)
  File "/usr/lib/python2.7/site-packages/celery/app/control.py", line 172, in revoke
    'signal': signal}, **kwargs)
  File "/usr/lib/python2.7/site-packages/celery/app/control.py", line 316, in broadcast
    limit, callback, channel=channel,
  File "/usr/lib/python2.7/site-packages/kombu/pidbox.py", line 283, in _broadcast
    chan = channel or self.connection.default_channel
  File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 756, in default_channel
    self.connection
  File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 741, in connection
    self._connection = self._establish_connection()
  File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 696, in _establish_connection
    conn = self.transport.establish_connection()
  File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 116, in establish_connection
    conn = self.Connection(**opts)
  File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 165, in __init__
    self.transport = self.Transport(host, connect_timeout, ssl)
  File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 186, in Transport
    return create_transport(host, connect_timeout, ssl)
  File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 299, in create_transport
    return TCPTransport(host, connect_timeout)
  File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 95, in __init__
    raise socket.error(last_err)
error: [Errno 111] Connection refused
```

**worker_shutdown** signal:

```
[2017-06-21 22:40:47,542: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.
Trying again in 2.00 seconds...

[2017-06-21 22:40:49,554: ERROR/MainProcess] consumer: Cannot connect to amqp://guest:**@127.0.0.1:5672//: [Errno 111] Connection refused.
Trying again in 4.00 seconds...

^C
worker: Hitting Ctrl+C again will terminate all running tasks!

worker: Warm shutdown (MainProcess)
[2017-06-21 22:40:52,860: WARNING/MainProcess] Traceback (most recent call last):
[2017-06-21 22:40:52,861: WARNING/MainProcess] File "/usr/lib64/python2.7/multiprocessing/util.py", line 274, in _run_finalizers
[2017-06-21 22:40:52,861: WARNING/MainProcess] finalizer()
[2017-06-21 22:40:52,861: WARNING/MainProcess] File "/usr/lib64/python2.7/multiprocessing/util.py", line 207, in __call__
[2017-06-21 22:40:52,862: WARNING/MainProcess] res = self._callback(*self._args, **self._kwargs)
[2017-06-21 22:40:52,862: WARNING/MainProcess] File "/usr/lib/python2.7/site-packages/celery/worker/__init__.py", line 201, in _send_worker_shutdown
[2017-06-21 22:40:52,862: WARNING/MainProcess] signals.worker_shutdown.send(sender=self)
[2017-06-21 22:40:52,862: WARNING/MainProcess] File "/usr/lib/python2.7/site-packages/celery/utils/dispatch/signal.py", line 166, in send
[2017-06-21 22:40:52,863: WARNING/MainProcess] response = receiver(signal=self, sender=sender, **named)
[2017-06-21 22:40:52,863: WARNING/MainProcess] File "/home/vagrant/devel/reproducer.py", line 20, in delete_worker
[2017-06-21 22:40:52,864: WARNING/MainProcess] controller.revoke(uuid4(), terminate=True)
[2017-06-21 22:40:52,864: WARNING/MainProcess] File "/usr/lib/python2.7/site-packages/celery/app/control.py", line 172, in revoke
[2017-06-21 22:40:52,864: WARNING/MainProcess] 'signal': signal}, **kwargs)
[2017-06-21 22:40:52,864: WARNING/MainProcess] File "/usr/lib/python2.7/site-packages/celery/app/control.py", line 316, in broadcast
[2017-06-21 22:40:52,865: WARNING/MainProcess] limit, callback, channel=channel,
[2017-06-21 22:40:52,865: WARNING/MainProcess] File "/usr/lib/python2.7/site-packages/kombu/pidbox.py", line 283, in _broadcast
[2017-06-21 22:40:52,865: WARNING/MainProcess] chan = channel or self.connection.default_channel
[2017-06-21 22:40:52,865: WARNING/MainProcess] File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 756, in default_channel
[2017-06-21 22:40:52,866: WARNING/MainProcess] self.connection
[2017-06-21 22:40:52,866: WARNING/MainProcess] File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 741, in connection
[2017-06-21 22:40:52,866: WARNING/MainProcess] self._connection = self._establish_connection()
[2017-06-21 22:40:52,866: WARNING/MainProcess] File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 696, in _establish_connection
[2017-06-21 22:40:52,867: WARNING/MainProcess] conn = self.transport.establish_connection()
[2017-06-21 22:40:52,867: WARNING/MainProcess] File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 116, in establish_connection
[2017-06-21 22:40:52,867: WARNING/MainProcess] conn = self.Connection(**opts)
[2017-06-21 22:40:52,867: WARNING/MainProcess] File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 165, in __init__
[2017-06-21 22:40:52,868: WARNING/MainProcess] self.transport = self.Transport(host, connect_timeout, ssl)
[2017-06-21 22:40:52,868: WARNING/MainProcess] File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 186, in Transport
[2017-06-21 22:40:52,868: WARNING/MainProcess] return create_transport(host, connect_timeout, ssl)
[2017-06-21 22:40:52,869: WARNING/MainProcess] File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 299, in create_transport
[2017-06-21 22:40:52,869: WARNING/MainProcess] return TCPTransport(host, connect_timeout)
[2017-06-21 22:40:52,869: WARNING/MainProcess] File "/usr/lib/python2.7/site-packages/amqp/transport.py", line 95, in __init__
[2017-06-21 22:40:52,869: WARNING/MainProcess] raise socket.error(last_err)
[2017-06-21 22:40:52,870: WARNING/MainProcess] error: [Errno 111] Connection refused
```


Discussion:
If the `revoke()` is being run by a worker, I expect it to have this wait-and-retry behavior.

Imagine that a webhandler, e.g. httpd, imports the Celery code and calls `revoke()`. In that case having the wait-and-retry behavior could be problematic because after some time, e.g. 500 seconds, the WSGI request will time-out. So revoke has different usages and I'm not sure if those usages affect the expected wait-and-retry behavior.

I'd like to hear some opinions from the current core maintainers about their thoughts on this expectation. I'm also looking for insight into why this isn't working given that our usage is running on the worker during its startup and shutdown signals.
@auvipy Do you or another core dev have an opinion on this? We are trying to determine if this is the intended behavior and we should implement a workaround or if this is expected to work and just not working correctly.
sorry for being late. I couldn't remember any reason for the exception, but I am on the side of inclusion if that increase dev experience.
@thedrow what you say?
I'm willing to accept a PR that adds a retry parameter to `signal.connect` defaulting to **false**. Other functionality such as backoff can also be implemented.
@thedrow That idea looks clear and would resolve the problem for us. Thanks.

Hopefully someone from the Pulp team can contribute a PR. We want to target the 4.0 branch right? We should also submit docs with it. Ask used to put together the release notes so we left that out of our commits. Do you want this PR to include a release note for it too?

Also we want to backport the feature to the 3.1 branch so that we can carry the patch downstream for 3.1. Will there be any more 3.1 releases?
no one except me got time or interest for 3.x branch. you should send against master first. after that things could be back ported.

Resolving patch:
diff --git a/celery/utils/dispatch/signal.py b/celery/utils/dispatch/signal.py
--- a/celery/utils/dispatch/signal.py
+++ b/celery/utils/dispatch/signal.py
@@ -5,10 +5,12 @@
 import threading
 import weakref
 import warnings
+from kombu.utils.functional import retry_over_time
 from celery.exceptions import CDeprecationWarning
 from celery.five import python_2_unicode_compatible, range, text_t
 from celery.local import PromiseProxy, Proxy
 from celery.utils.functional import fun_accepts_kwargs
+from celery.utils.time import humanize_seconds
 from celery.utils.log import get_logger
 try:
     from weakref import WeakMethod
@@ -36,6 +38,10 @@ def _make_id(target):  # pragma: no cover
 
 NO_RECEIVERS = object()
 
+RECEIVER_RETRY_ERROR = """\
+Could not process signal receiver %(receiver)s. Retrying %(when)s...\
+"""
+
 
 @python_2_unicode_compatible
 class Signal(object):  # pragma: no cover
@@ -103,12 +109,49 @@ def connect(self, *args, **kwargs):
             dispatch_uid (Hashable): An identifier used to uniquely identify a
                 particular instance of a receiver.  This will usually be a
                 string, though it may be anything hashable.
+
+            retry (bool): If the signal receiver raises an exception
+                (e.g. ConnectionError), the receiver will be retried until it
+                runs successfully. A strong ref to the receiver will be stored
+                and the `weak` option will be ignored.
         """
-        def _handle_options(sender=None, weak=True, dispatch_uid=None):
+        def _handle_options(sender=None, weak=True, dispatch_uid=None,
+                            retry=False):
 
             def _connect_signal(fun):
-                self._connect_signal(fun, sender, weak, dispatch_uid)
+
+                options = {'dispatch_uid': dispatch_uid,
+                           'weak': weak}
+
+                def _retry_receiver(retry_fun):
+
+                    def _try_receiver_over_time(*args, **kwargs):
+                        def on_error(exc, intervals, retries):
+                            interval = next(intervals)
+                            err_msg = RECEIVER_RETRY_ERROR % \
+                                {'receiver': retry_fun,
+                                 'when': humanize_seconds(interval, 'in', ' ')}
+                            logger.error(err_msg)
+                            return interval
+
+                        return retry_over_time(retry_fun, Exception, args,
+                                               kwargs, on_error)
+
+                    return _try_receiver_over_time
+
+                if retry:
+                    options['weak'] = False
+                    if not dispatch_uid:
+                        # if there's no dispatch_uid then we need to set the
+                        # dispatch uid to the original func id so we can look
+                        # it up later with the original func id
+                        options['dispatch_uid'] = _make_id(fun)
+                    fun = _retry_receiver(fun)
+
+                self._connect_signal(fun, sender, options['weak'],
+                                     options['dispatch_uid'])
                 return fun
+
             return _connect_signal
 
         if args and callable(args[0]):
@@ -158,6 +201,7 @@ def _connect_signal(self, receiver, sender, weak, dispatch_uid):
             else:
                 self.receivers.append((lookup_key, receiver))
             self.sender_receivers_cache.clear()
+
         return receiver
 
     def disconnect(self, receiver=None, sender=None, weak=None,


Test patch:
diff --git a/t/unit/utils/test_dispatcher.py b/t/unit/utils/test_dispatcher.py
--- a/t/unit/utils/test_dispatcher.py
+++ b/t/unit/utils/test_dispatcher.py
@@ -143,3 +143,32 @@ def test_disconnection(self):
         finally:
             a_signal.disconnect(receiver_3)
         self._testIsClean(a_signal)
+
+    def test_retry(self):
+
+        class non_local:
+            counter = 1
+
+        def succeeds_eventually(val, **kwargs):
+            non_local.counter += 1
+            if non_local.counter < 3:
+                raise ValueError('this')
+
+            return val
+
+        a_signal.connect(succeeds_eventually, sender=self, retry=True)
+        try:
+            result = a_signal.send(sender=self, val='test')
+            assert non_local.counter == 3
+            assert result[0][1] == 'test'
+        finally:
+            a_signal.disconnect(succeeds_eventually, sender=self)
+        self._testIsClean(a_signal)
+
+    def test_retry_with_dispatch_uid(self):
+        uid = 'abc123'
+        a_signal.connect(receiver_1_arg, sender=self, retry=True,
+                         dispatch_uid=uid)
+        assert a_signal.receivers[0][0][0] == uid
+        a_signal.disconnect(receiver_1_arg, sender=self, dispatch_uid=uid)
+        self._testIsClean(a_signal)