Repository: celery/celery

Issue:
ResultConsumer greenlet race condition
When I tried to get result from task inside greenlet, I've got an exception:

```
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/gevent/greenlet.py", line 534, in run
    result = self._run(*self.args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/celery/backends/async.py", line 83, in run
    self.result_consumer.drain_events(timeout=1)
  File "/usr/local/lib/python2.7/dist-packages/celery/backends/redis.py", line 69, in drain_events
    m = self._pubsub.get_message(timeout=timeout)
AttributeError: 'NoneType' object has no attribute 'get_message'
<Greenlet at 0x7efd9d8ba550: <bound method geventDrainer.run of <celery.backends.async.geventDrainer object at 0x7efd9d99f550>>> failed with AttributeError
```

Celery version:

```
software -> celery:4.0.2 (latentcall) kombu:4.0.2 py:2.7.12
                   billiard:3.5.0.2 redis:2.10.5
platform -> system:Linux arch:64bit, ELF imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:redis results:redis
```

After some recon, I've found that error occurs, when `threading` module is monkey-patched by `gevent.monkey`. It seems that attribute `_pubsub` from `celery.backends.async.ResultConsumer` is initialized a bit too late. Bug is triggered by racing `greenletDrainer.start()` and `ResultConsumer.start()` initializers.

```python
    def add_pending_result(self, result, weak=False, start_drainer=True):
        if start_drainer:
            # Spawns greenlet and starts to drain events
            self.result_consumer.drainer.start()
        try:
            self._maybe_resolve_from_buffer(result)
        except Empty:
            # Initializes pubsub needed by drainer
            self._add_pending_result(result.id, result, weak=weak)
        return result
```

```python
class greenletDrainer(Drainer):
    #...

    def run(self):
        self._started.set()
        while not self._stopped.is_set():
            try:
                self.result_consumer.drain_events(timeout=1)
            except socket.timeout:
                pass
        self._shutdown.set()

    # self.result_consumer.drainer.start()
    def start(self):
        if not self._started.is_set():
            self._g = self.spawn(self.run)
            # Switches immediately to self.run
            self._started.wait()
```

Issue related with #3452.

Filtered Python trace and way to reproduce a bug can be found [in this Gist](https://gist.github.com/psrok1/8dc27d3cdf367573183fc3f1e5524293)




Discussion:
Bypass:
```python
# Manually initialize consumer
celery_app.backend.result_consumer.start("")

def fetch(uuid):
    res = AsyncResult(uuid, app=celery_app).get()
    print res
```
can you send a patch with test to fix the issue?

Resolving patch:
diff --git a/celery/backends/redis.py b/celery/backends/redis.py
--- a/celery/backends/redis.py
+++ b/celery/backends/redis.py
@@ -2,6 +2,8 @@
 """Redis result store backend."""
 from __future__ import absolute_import, unicode_literals
 
+import time
+
 from functools import partial
 from ssl import CERT_NONE, CERT_OPTIONAL, CERT_REQUIRED
 
@@ -117,9 +119,12 @@ def stop(self):
             self._pubsub.close()
 
     def drain_events(self, timeout=None):
-        message = self._pubsub.get_message(timeout=timeout)
-        if message and message['type'] == 'message':
-            self.on_state_change(self._decode_result(message['data']), message)
+        if self._pubsub:
+            message = self._pubsub.get_message(timeout=timeout)
+            if message and message['type'] == 'message':
+                self.on_state_change(self._decode_result(message['data']), message)
+        elif timeout:
+            time.sleep(timeout)
 
     def consume_from(self, task_id):
         if self._pubsub is None:


Test patch:
diff --git a/t/unit/backends/test_redis.py b/t/unit/backends/test_redis.py
--- a/t/unit/backends/test_redis.py
+++ b/t/unit/backends/test_redis.py
@@ -189,6 +189,11 @@ def test_on_state_change(self, parent_method, cancel_for):
         parent_method.assert_called_once_with(meta, message)
         cancel_for.assert_not_called()
 
+    def test_drain_events_before_start(self):
+        consumer = self.get_consumer()
+        # drain_events shouldn't crash when called before start
+        consumer.drain_events(0.001)
+
 
 class test_RedisBackend:
     def get_backend(self):
diff --git a/t/unit/backends/test_rpc.py b/t/unit/backends/test_rpc.py
--- a/t/unit/backends/test_rpc.py
+++ b/t/unit/backends/test_rpc.py
@@ -8,6 +8,19 @@
 from celery.backends.rpc import RPCBackend
 
 
+class test_RPCResultConsumer:
+    def get_backend(self):
+        return RPCBackend(app=self.app)
+
+    def get_consumer(self):
+        return self.get_backend().result_consumer
+
+    def test_drain_events_before_start(self):
+        consumer = self.get_consumer()
+        # drain_events shouldn't crash when called before start
+        consumer.drain_events(0.001)
+
+
 class test_RPCBackend:
 
     def setup(self):