ask/celery — Issue #597: All workers unpredictably hang
Author: lsemel
Created: 2012-01-22 04:16:25+00:00
State: open

After running Celery for a few days, my tasks unpredictably hang.

In the log, I don't see workers exiting, I see notices of the workers having received or completed previous tasks, then nothing. Often the last log message indicates that the MainProcess got a task from the broker, and I see log messages form the individual tasks indicating that tasks were in progress but just stopped dead. The tasks quit at different points, so they're not stuck waiting on some resource.

Using celeryctl inspect, I see no active tasks, but a long list of reserved tasks. Rabbitmq, which I am using as my queue, seems to be up and working fine. Celeryev shows "Info: events:2 tasks:0 workers:1/1" even though there are 4 worker processes running. celeryctl status reports "OK, 1 node online" as expected.

In gdb the workers show with a stacktrace like this:

(gdb) bt
#0 0xf57fe416 in __kernel_vsyscall ()
#1 0xb775d245 in sem_wait@@GLIBC_2.1 () from /lib/tls/i686/cmov/libpthread.so.0
#2 0xb72137c6 in semlock_acquire (self=0xa01f3c0, args=0xb73d702c, kwds=0x0) at /build/buildd/python2.6-2.6.5/Modules/_multiprocessing/semaphore.c:310
#3 0x080e0a21 in PyEval_EvalFrameEx ()
#4 0x080e2807 in PyEval_EvalCodeEx ()
...

My versions are
python=2.6.5
celery==2.2.6
django-celery==2.2.4