Repository: celery/celery

Issue:
Consumer does not shutdown properly when embedded in gevent application
Consider `blueprint.state != CLOSE` in https://github.com/celery/celery/blob/master/celery/worker/consumer/consumer.py#L308 and
https://github.com/celery/celery/blob/master/celery/worker/consumer/consumer.py#L327

and the logic in https://github.com/celery/celery/blob/master/celery/bootsteps.py#L169 and https://github.com/celery/celery/blob/master/celery/bootsteps.py#L178

With gevent the total order is enforced by virtue of running in a loop. Therefore the problem is consistently reproducible: when `WorkerController` is stopped, the `blueprint.state` transitions from `CLOSE` to `TERMINATE`. The execution returns to `Consumer.start()` iterates in the while loop seeing that `blueprint.state != CLOSE` when in fact it is `TERMINATE` and restarts the consumer yet again preventing `WorkerController` stop.

This produces the following peculiar output:
```
KeyboardInterrupt
Sun Jan  8 07:34:49 2017
DEBUG 2017-01-08 07:34:55,691 bootsteps 21273 140206893775912 | Worker: Closing Pool...
DEBUG 2017-01-08 07:34:55,693 bootsteps 21273 140206893775912 | Worker: stopping <step: Tasks>
DEBUG 2017-01-08 07:34:55,693 tasks 21273 140206893775912 Canceling task consumer...
DEBUG 2017-01-08 07:34:55,695 bootsteps 21273 140206893775912 | Worker: stopping <step: event loop>
DEBUG 2017-01-08 07:34:55,696 bootsteps 21273 140206893775912 | Worker: Closing Consumer...
DEBUG 2017-01-08 07:35:22,008 bootsteps 21273 140206893775912 | Worker: Stopping Consumer...
DEBUG 2017-01-08 07:35:36,808 bootsteps 21273 140206893775912 | Consumer: Closing Connection...
DEBUG 2017-01-08 07:35:36,808 bootsteps 21273 140206893775912 | Consumer: Closing Events...
DEBUG 2017-01-08 07:35:36,809 bootsteps 21273 140206893775912 | Consumer: Closing Heart...
DEBUG 2017-01-08 07:35:36,810 bootsteps 21273 140206893775912 | Consumer: Closing Tasks...
DEBUG 2017-01-08 07:35:36,811 bootsteps 21273 140206893775912 | Consumer: Closing event loop...
DEBUG 2017-01-08 07:35:39,092 bootsteps 21273 140206893775912 | Consumer: Stopping event loop...
DEBUG 2017-01-08 07:35:39,092 bootsteps 21273 140206893775912 | Consumer: Stopping Tasks...
DEBUG 2017-01-08 07:35:39,093 tasks 21273 140206893775912 Canceling task consumer...
DEBUG 2017-01-08 07:35:39,093 bootsteps 21273 140206893775912 | Consumer: Stopping Heart...
DEBUG 2017-01-08 07:35:39,094 bootsteps 21273 140206893775912 | Consumer: Stopping Events...
DEBUG 2017-01-08 07:35:39,094 bootsteps 21273 140206893775912 | Consumer: Stopping Connection...
DEBUG 2017-01-08 07:36:06,159 bootsteps 21273 140206893775912 | Worker: Stopping Pool...
DEBUG 2017-01-08 07:36:11,245 bootsteps 21273 140206893775912 | Consumer: Shutdown Tasks...
DEBUG 2017-01-08 07:36:11,246 tasks 21273 140206893775912 Canceling task consumer...
DEBUG 2017-01-08 07:36:11,247 tasks 21273 140206893775912 Closing consumer channel...
DEBUG 2017-01-08 07:36:11,247 bootsteps 21273 140206893775912 | Consumer: Shutdown Heart...
DEBUG 2017-01-08 07:36:11,248 bootsteps 21273 140206893775912 | Consumer: Shutdown Events...
DEBUG 2017-01-08 07:36:11,249 bootsteps 21273 140206893775912 | Consumer: Shutdown Connection...
DEBUG 2017-01-08 07:36:21,035 bootsteps 21273 140206893774392 ^-- substep ok
DEBUG 2017-01-08 07:38:53,621 bootsteps 21273 140206893774392 | Consumer: Starting Connection
INFO 2017-01-08 07:38:53,643 connection 21273 140206893774392 Connected to django://localhost//
DEBUG 2017-01-08 07:38:53,644 bootsteps 21273 140206893774392 ^-- substep ok
DEBUG 2017-01-08 07:38:53,645 bootsteps 21273 140206893774392 | Consumer: Starting Events
DEBUG 2017-01-08 07:38:53,657 bootsteps 21273 140206893774392 ^-- substep ok
DEBUG 2017-01-08 07:38:53,657 bootsteps 21273 140206893774392 | Consumer: Starting Heart
DEBUG 2017-01-08 07:38:53,658 bootsteps 21273 140206893774392 ^-- substep ok
DEBUG 2017-01-08 07:38:53,658 bootsteps 21273 140206893774392 | Consumer: Starting Tasks
DEBUG 2017-01-08 07:38:53,671 bootsteps 21273 140206893774392 ^-- substep ok
DEBUG 2017-01-08 07:38:53,672 bootsteps 21273 140206893774392 | Consumer: Starting event loop
```

The fix is trivial, PR will be submitted.


Discussion:

Resolving patch:
diff --git a/celery/worker/consumer/consumer.py b/celery/worker/consumer/consumer.py
--- a/celery/worker/consumer/consumer.py
+++ b/celery/worker/consumer/consumer.py
@@ -42,6 +42,8 @@
 __all__ = ['Consumer', 'Evloop', 'dump_body']
 
 CLOSE = bootsteps.CLOSE
+TERMINATE = bootsteps.TERMINATE
+STOP_CONDITIONS = {CLOSE, TERMINATE}
 logger = get_logger(__name__)
 debug, info, warn, error, crit = (logger.debug, logger.info, logger.warning,
                                   logger.error, logger.critical)
@@ -305,7 +307,7 @@ def _limit_task(self, request, bucket, tokens):
 
     def start(self):
         blueprint = self.blueprint
-        while blueprint.state != CLOSE:
+        while blueprint.state not in STOP_CONDITIONS:
             maybe_shutdown()
             if self.restart_count:
                 try:
@@ -324,7 +326,7 @@ def start(self):
                 if isinstance(exc, OSError) and exc.errno == errno.EMFILE:
                     raise  # Too many open files
                 maybe_shutdown()
-                if blueprint.state != CLOSE:
+                if blueprint.state not in STOP_CONDITIONS:
                     if self.connection:
                         self.on_connection_error_after_connected(exc)
                     else:


Test patch:
diff --git a/t/unit/worker/test_consumer.py b/t/unit/worker/test_consumer.py
--- a/t/unit/worker/test_consumer.py
+++ b/t/unit/worker/test_consumer.py
@@ -10,7 +10,8 @@
 from billiard.exceptions import RestartFreqExceeded
 
 from celery.worker.consumer.agent import Agent
-from celery.worker.consumer.consumer import CLOSE, Consumer, dump_body
+from celery.worker.consumer.consumer import (CLOSE, TERMINATE,
+                                             Consumer, dump_body)
 from celery.worker.consumer.gossip import Gossip
 from celery.worker.consumer.heart import Heart
 from celery.worker.consumer.mingle import Mingle
@@ -153,6 +154,35 @@ def se(*args, **kwargs):
             c.start()
             sleep.assert_called_with(1)
 
+    def test_do_not_restart_when_closed(self):
+        c = self.get_consumer()
+
+        c.blueprint.state = None
+
+        def bp_start(*args, **kwargs):
+            c.blueprint.state = CLOSE
+
+        c.blueprint.start.side_effect = bp_start
+        with patch('celery.worker.consumer.consumer.sleep'):
+            c.start()
+
+        c.blueprint.start.assert_called_once_with(c)
+
+    def test_do_not_restart_when_terminated(self):
+        c = self.get_consumer()
+
+        c.blueprint.state = None
+
+        def bp_start(*args, **kwargs):
+            c.blueprint.state = TERMINATE
+
+        c.blueprint.start.side_effect = bp_start
+
+        with patch('celery.worker.consumer.consumer.sleep'):
+            c.start()
+
+        c.blueprint.start.assert_called_once_with(c)
+
     def test_no_retry_raises_error(self):
         self.app.conf.broker_connection_retry = False
         c = self.get_consumer()