ask/celery — Issue #572: worker processes stuck in FUTEX_WAIT_PRIVATE
Author: matthewlmcclure
Created: 2011-12-18 21:28:47+00:00
State: open

I'm using:

celery 2.2.6
django-celery 2.2.4
kombu 1.1.5
RabbitMQ 2.2.0

Over the past few days, I've seen the following symtoms. I'm unsure what the causal relationships are.

* Of the 8 celery worker processes, some portion stop consuming CPU.
* strace -p indicates the process is stuck in FUTEX_WAIT_PRIVATE.
* The message queue depth in RabbitMQ rises sharply into the thousands of messages.
* Killing the stuck workers allows supervisor to restart another instance.
* After killing stuck workers, their replacement workers consume CPU, and the message queue depth falls.
* beam.smp consumes > 1g resident memory on the RabbitMQ host machine.
* Load average on RabbitMQ's host machine rises to 2-3x typical levels, with beam.smp consuming 2-3x the CPU it typically does.
* There are tens of thousands of queues in the output of rabbitmqctl list_queues with 1 message.
* Stopping RabbitMQ with rabbitmqctl stop allows supervisor to start a new instance that at least initially uses far less memory: tens or low hundreds of megabytes.
* After stopping/restarting RabbitMQ, load average on the host machine returns to typical levels.

At least superficially, there are some similarities between what I see and the observations in https://github.com/ask/celery/issues/417

The main questions I'd like to answer are:

* What causes the celery workers to get stuck in FUTEX_WAIT_PRIVATE?
* How can I prevent it, or how can I make the system heal itself from that state?
* What causes beam.smp to consume more memory than usual? More CPU than usual?
* How can I prevent beam.smp from doing so?
* Are the symptoms of beam.smp the cause of the celery workers' symptoms? Vice-versa? Do they share a single different cause? Do they have different causes altogether?

I'd appreciate any ideas of where to look next.