Issue 1:
Every couple of weeks our worker stops processing tasks. Nothing amiss in the celery logs, just one minute they are ticking along nicely, the next there are no more log entries at all. All the relevant processes are still running. Restarting the worker works fine and everything starts up again.

Running the most recent version:

celery==3.1.17
kombu==3.0.24
billiard==3.3.0.19

Using with Django, with Redis as broker. The only thing we've set is CELERY_RESULT_BACKEND, BROKER_URL (to the same value), and BROKER_TRANSPORT_OPTIONS (setting custom priority_steps) -- no other changes from the defaults there. When starting a worker we set --maxtasksperchild=100, --concurrency=16 and set logging and select a queue, but that's all.

Running celery status does indeed respond, and shows 2 nodes are online (we run two workers with the above settings -- only 1 has ever "died", the one that gets significantly more traffic).

Wondering what kind of information I should collect next time it happens so that I can start tracking this down (I can't reproduce on a test setup so far). Cheers.

Issue 2:
Celery workers become stuck/deadlocked when using Redis broker in Celery 3.1.17.

Reproduced in 3.1.0 and 3.1.16 as well.

Issue does not occur in current Celery master (3.2.0a2).

Issue does not occur with RabbitMQ as broker.

Reproducable on Linux and Mac OS.

Environment

Scientific Linux release 6.5 or Mac OS 10.10.2

Python 2.7.9

Redis 2.8.9

Celery 3.1.0, 3.1.16, 3.1.17

virtualenv 12.0.5

software -> celery:3.1.17 (Cipater) kombu:3.0.24 py:2.7.9
billiard:3.3.0.19 redis:2.10.3
platform -> system:Linux arch:64bit, ELF imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:redis results:disabled

BROKER_URL: 'redis://localhost:6380/0'

redis.conf

# close the connection after a client is idle for N seconds
timeout 0

celeryconfig.py

BROKER_URL = 'redis://localhost:6380/0'

tasks.py

from celery import Celery
import celeryconfig

import time

app = Celery('tasks')
app.config_from_object(celeryconfig)

@app.task
def myTask(id):
return id

test_client.py

from multiprocessing import Pool
from tasks import myTask

def run(task_id):
myTask.delay(1)

num_tasks = 20000
pool = Pool(processes=4)
pool.map(run, xrange(num_tasks))
pool.close()
pool.join()

Reproducing

Ran test_client.py to submit 20K tasks.

Started two concurrent workers (-Ofair doesn't make a difference):

celery -A tasks worker --loglevel=DEBUG --concurrency=3 [-Ofair]

After ~10-20K of processed tasks, workers stuck/deadlocked.

[2015-01-21 17:40:31,304: DEBUG/MainProcess] Task accepted: tasks.regulatorTask[6d2be28f-4d97-43b3-84dd-ad28b3a9d0f7] pid:13242
[2015-01-21 17:40:31,305: DEBUG/MainProcess] Task accepted: tasks.regulatorTask[6d798559-3e1e-4fe8-a813-065696d8a1df] pid:13243
[2015-01-21 17:40:31,306: INFO/MainProcess] Received task: tasks.regulatorTask[6adbe350-2d5e-4a92-89d9-4bbd63973aa6]
[2015-01-21 17:40:31,306: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f3436baf848> (args:('tasks.regulatorTask', '6adbe350-2d5e-4a92-89d9-4bbd63973aa6', (1,), {}, {'utc': True, u'is_eager': False, 'chord': None, u'group': None, 'args': (1,), 'retries': 0, u'delivery_info': {u'priority': 0, u'redelivered': None, u'routing_key': u'celery', u'exchange': u'celery'}, 'expires': None, u'hostname': 'celery@dev6-cig-c53d391f.ewr01.tumblr.net', 'task': 'tasks.regulatorTask', 'callbacks': None, u'correlation_id': u'6adbe350-2d5e-4a92-89d9-4bbd63973aa6', 'errbacks': None, 'timelimit': (None, None), 'taskset': None, 'kwargs': {}, 'eta': None, u'reply_to': u'7d9537ab-0a15-34eb-afa2-aafc60e63b1a', 'id': '6adbe350-2d5e-4a92-89d9-4bbd63973aa6', u'headers': {}}) kwargs:{})
[2015-01-21 17:40:31,307: INFO/MainProcess] Task tasks.regulatorTask[6d2be28f-4d97-43b3-84dd-ad28b3a9d0f7] succeeded in 0.00572886499867s: 1
[2015-01-21 17:40:31,308: INFO/MainProcess] Task tasks.regulatorTask[6d798559-3e1e-4fe8-a813-065696d8a1df] succeeded in 0.00644170500163s: 1

strace indicates workers blocking on I/O read from Redis

Process 13241 attached - interrupt to quit
read(13,

Process 13242 attached - interrupt to quit
read(17,

Process 13243 attached - interrupt to quit
read(9,

Notables

adding a connection timeout in Redis will cause deadlock to release after timeout period

specifying a CELERYD_TASK_TIME_LIMIT does not prevent/release the deadlock

single worker execution (--concurrency=1) works properly