Repository: celery/celery

Issue:
"assert proc.outqR_fd in hub.readers" AssertionError
```
[2014-01-13 15:06:53,047] pid=33970/MainProcess - ERROR    - celery.worker - Unrecoverable error: AssertionError()
Traceback (most recent call last):
  File "/home/ionel/projects/core/.ve/local/lib/python2.7/site-packages/celery/worker/__init__.py", line 206, in start
    self.blueprint.start(self)
  File "/home/ionel/projects/core/.ve/local/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
    step.start(parent)
  File "/home/ionel/projects/core/.ve/local/lib/python2.7/site-packages/celery/bootsteps.py", line 373, in start
    return self.obj.start()
  File "/home/ionel/projects/core/.ve/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 270, in start
    blueprint.start(self)
  File "/home/ionel/projects/core/.ve/local/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
    step.start(parent)
  File "/home/ionel/projects/core/.ve/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 786, in start
    c.loop(*c.loop_args())
  File "/home/ionel/projects/core/.ve/local/lib/python2.7/site-packages/celery/worker/loops.py", line 71, in asynloop
    next(loop)
  File "/home/ionel/projects/core/.ve/local/lib/python2.7/site-packages/kombu/async/hub.py", line 288, in create_loop
    poll_timeout = fire_timers(propagate=propagate) if scheduled else 1
  File "/home/ionel/projects/core/.ve/local/lib/python2.7/site-packages/kombu/async/hub.py", line 151, in fire_timers
    entry()
  File "/home/ionel/projects/core/.ve/local/lib/python2.7/site-packages/kombu/async/timer.py", line 64, in __call__
    return self.fun(*self.args, **self.kwargs)
  File "/home/ionel/projects/core/.ve/local/lib/python2.7/site-packages/celery/concurrency/asynpool.py", line 504, in verify_process_alive
    assert proc.outqR_fd in hub.readers
AssertionError
[2014-01-13 15:06:53,048] pid=33970/MainProcess - DEBUG    - celery.bootsteps - | Worker: Closing Hub...
[2014-01-13 15:06:53,048] pid=33970/MainProcess - DEBUG    - celery.bootsteps - | Worker: Closing Pool...
[2014-01-13 15:06:53,048] pid=33970/MainProcess - DEBUG    - celery.bootsteps - | Worker: Closing Consumer...
[2014-01-13 15:06:53,048] pid=33970/MainProcess - DEBUG    - celery.bootsteps - | Worker: Stopping Consumer...
[2014-01-13 15:06:53,048] pid=33970/MainProcess - DEBUG    - celery.bootsteps - | Consumer: Closing Connection...
[2014-01-13 15:06:53,049] pid=33970/MainProcess - DEBUG    - celery.bootsteps - | Consumer: Closing Events...
[2014-01-13 15:06:53,049] pid=33970/MainProcess - DEBUG    - celery.bootsteps - | Consumer: Closing Mingle...
[2014-01-13 15:06:53,049] pid=33970/MainProcess - DEBUG    - celery.bootsteps - | Consumer: Closing Tasks...
[2014-01-13 15:06:53,049] pid=33970/MainProcess - DEBUG    - celery.bootsteps - | Consumer: Closing Control...
[2014-01-13 15:06:53,049] pid=33970/MainProcess - DEBUG    - celery.bootsteps - | Consumer: Closing Gossip...
[2014-01-13 15:06:53,049] pid=33970/MainProcess - DEBUG    - celery.bootsteps - | Consumer: Closing Heart...
[2014-01-13 15:06:53,049] pid=33970/MainProcess - DEBUG    - celery.bootsteps - | Consumer: Closing event loop...
[2014-01-13 15:06:53,049] pid=33970/MainProcess - DEBUG    - celery.bootsteps - | Consumer: Stopping event loop...
[2014-01-13 15:06:53,049] pid=33970/MainProcess - DEBUG    - celery.bootsteps - | Consumer: Stopping Heart...
[2014-01-13 15:06:53,049] pid=33970/MainProcess - DEBUG    - celery.bootsteps - | Consumer: Stopping Gossip...
[2014-01-13 15:06:53,050] pid=33970/MainProcess - DEBUG    - celery.bootsteps - | Consumer: Stopping Control...
```

I don't have a isolated testcase yet. Does the cause look like something obvious ?



Discussion:
Tested with:
- celery/kombu@ffa90945bf06ba8b9269b4a36019baad0ac57793
- celery/billiard@c29c4f7adbd0f7f4544c05fb9777800616e89d2f
- celery/celery@ceaf7aba36eae78af852eb5ca703c81091b52f23

[I too](https://github.com/celery/kombu/issues/305) am getting a similar issue. Which transport are you using?

It was redis. I will post more details when I get the chance

I am getting this issues with rabbitmq/amqp.

I'm getting it too, on Redis backend. Celery Beat continues, worker restarts but seems to be idle (no output from tasks in log, unlike when I stop entire instance, do flushall on redis and start celery again, then it starts working again).

The problem code is in https://github.com/celery/celery/blob/master/celery/concurrency/asynpool.py#L501

Do you guys use CELERYD_MAX_TASKS_PER_CHILD with low value ?

Yes! In fact I set it to 1, otherwise I had massive memory leaks under heavy loads (lots of workers, big tasks, not releasing memory to OS causing memory use creeping up and then OOM killer incoming) etc.

We have it set to 1000. 

Yep it is set to 2500 for us.


Resolving patch:
diff --git a/celery/concurrency/asynpool.py b/celery/concurrency/asynpool.py
--- a/celery/concurrency/asynpool.py
+++ b/celery/concurrency/asynpool.py
@@ -501,7 +501,7 @@ def verify_process_alive(proc):
             if proc._is_alive() and proc in waiting_to_start:
                 assert proc.outqR_fd in fileno_to_outq
                 assert fileno_to_outq[proc.outqR_fd] is proc
-                assert proc.outqR_fd in hub.readers
+                assert proc.outqR_fd in hub.readers, "%s.outqR_fd=%s not in hub.readers !" % (proc, proc.outqR_fd)
                 error('Timed out waiting for UP message from %r', proc)
                 os.kill(proc.pid, 9)
 
@@ -570,6 +570,15 @@ def on_process_down(proc):
             if inq:
                 busy_workers.discard(inq)
             hub_remove(proc.sentinel)
+            waiting_to_start.discard(proc)
+            self._active_writes.discard(proc.inqW_fd)
+            hub_remove(proc.inqW_fd)
+            hub_remove(proc.outqR_fd)
+            if proc.synqR_fd:
+                hub_remove(proc.synqR_fd)
+            if proc.synqW_fd:
+                self._active_writes.discard(proc.synqW_fd)
+                hub_remove(proc.synqW_fd)
         self.on_process_down = on_process_down
 
     def _create_write_handlers(self, hub,
@@ -966,7 +975,7 @@ def on_process_alive(self, pid):
         try:
             proc = next(w for w in self._pool if w.pid == pid)
         except StopIteration:
-            # process already exited :(  this will be handled elsewhere.
+            logger.warning("process with pid=%s already exited :( - handling this elsewhere ...", pid)
             return
         assert proc.inqW_fd not in self._fileno_to_inq
         assert proc.inqW_fd not in self._all_inqueues


Test patch: