Issue 1:
Celery workers become stuck/deadlocked when using Redis broker in Celery 3.1.17.

Reproduced in 3.1.0 and 3.1.16 as well.

Issue does not occur in current Celery master (3.2.0a2).

Issue does not occur with RabbitMQ as broker.

Reproducable on Linux and Mac OS.

Environment

Scientific Linux release 6.5 or Mac OS 10.10.2

Python 2.7.9

Redis 2.8.9

Celery 3.1.0, 3.1.16, 3.1.17

virtualenv 12.0.5

software -> celery:3.1.17 (Cipater) kombu:3.0.24 py:2.7.9
billiard:3.3.0.19 redis:2.10.3
platform -> system:Linux arch:64bit, ELF imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:redis results:disabled

BROKER_URL: 'redis://localhost:6380/0'

redis.conf

# close the connection after a client is idle for N seconds
timeout 0

celeryconfig.py

BROKER_URL = 'redis://localhost:6380/0'

tasks.py

from celery import Celery
import celeryconfig

import time

app = Celery('tasks')
app.config_from_object(celeryconfig)

@app.task
def myTask(id):
return id

test_client.py

from multiprocessing import Pool
from tasks import myTask

def run(task_id):
myTask.delay(1)

num_tasks = 20000
pool = Pool(processes=4)
pool.map(run, xrange(num_tasks))
pool.close()
pool.join()

Reproducing

Ran test_client.py to submit 20K tasks.

Started two concurrent workers (-Ofair doesn't make a difference):

celery -A tasks worker --loglevel=DEBUG --concurrency=3 [-Ofair]

After ~10-20K of processed tasks, workers stuck/deadlocked.

[2015-01-21 17:40:31,304: DEBUG/MainProcess] Task accepted: tasks.regulatorTask[6d2be28f-4d97-43b3-84dd-ad28b3a9d0f7] pid:13242
[2015-01-21 17:40:31,305: DEBUG/MainProcess] Task accepted: tasks.regulatorTask[6d798559-3e1e-4fe8-a813-065696d8a1df] pid:13243
[2015-01-21 17:40:31,306: INFO/MainProcess] Received task: tasks.regulatorTask[6adbe350-2d5e-4a92-89d9-4bbd63973aa6]
[2015-01-21 17:40:31,306: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f3436baf848> (args:('tasks.regulatorTask', '6adbe350-2d5e-4a92-89d9-4bbd63973aa6', (1,), {}, {'utc': True, u'is_eager': False, 'chord': None, u'group': None, 'args': (1,), 'retries': 0, u'delivery_info': {u'priority': 0, u'redelivered': None, u'routing_key': u'celery', u'exchange': u'celery'}, 'expires': None, u'hostname': 'celery@dev6-cig-c53d391f.ewr01.tumblr.net', 'task': 'tasks.regulatorTask', 'callbacks': None, u'correlation_id': u'6adbe350-2d5e-4a92-89d9-4bbd63973aa6', 'errbacks': None, 'timelimit': (None, None), 'taskset': None, 'kwargs': {}, 'eta': None, u'reply_to': u'7d9537ab-0a15-34eb-afa2-aafc60e63b1a', 'id': '6adbe350-2d5e-4a92-89d9-4bbd63973aa6', u'headers': {}}) kwargs:{})
[2015-01-21 17:40:31,307: INFO/MainProcess] Task tasks.regulatorTask[6d2be28f-4d97-43b3-84dd-ad28b3a9d0f7] succeeded in 0.00572886499867s: 1
[2015-01-21 17:40:31,308: INFO/MainProcess] Task tasks.regulatorTask[6d798559-3e1e-4fe8-a813-065696d8a1df] succeeded in 0.00644170500163s: 1

strace indicates workers blocking on I/O read from Redis

Process 13241 attached - interrupt to quit
read(13,

Process 13242 attached - interrupt to quit
read(17,

Process 13243 attached - interrupt to quit
read(9,

Notables

adding a connection timeout in Redis will cause deadlock to release after timeout period

specifying a CELERYD_TASK_TIME_LIMIT does not prevent/release the deadlock

single worker execution (--concurrency=1) works properly

Issue 2:
So I have been battling this issue for a few days now but it's really hard to figure out exactly what causes it. I had a repro for a day on an amazon machine but unfortunately it was a spot instance and I no longer have access to it. I tried to make another repro on the weekend and failed. Here is what I'm observing:

two amazon machines (does not matter if in vpc or not)

install redis on one, celery on the other

run celery with four prefork workers

spam the queue with lots of tasks

After some period of time everything will lock up. Ways I have seen this recover:

restart celery

CLIENT KILL from the redis server (this will trigger some epoll errors).

We have a customer who also seems to have a related issue but it recovers after 30 minutes. I have not have a way to reproduce this issue, but it might be related to the client timeout on redis.

Things I know other than that:

strace: when it hangs, it's trying to read from a redis socket.

the read is _read_from_socket in SocketBuffer in py-redis

changing timeouts on the redis side changes nothing

enabling keepalives changes nothing

a dummy python script that does BLPOP in a loop against the redis server with a second script that puts items into the queue does not get stuck.

I have a few theories on what this could be but I'm honestly at the end of my understanding of the issue. What I see is that kombu internally catches down random epoll errors and when I add some prints there I can see that I get epoll errors even though the socket seems alive and happy. I'm wondering if it could be possible that for some reason wrong fds are attempted to be registered with that thing.

Versions used:

kombu==3.0.26

celery==3.1.18

redis==2.10.3

Redis server: 3.0.4

Happens with at least ubuntu 14.04

I have since tried to reproduce this again on another amazon machine and it does not appear to happen. I'm not entirely sure what triggers it but it seems like the socket py-redis tries to read from is not what it thinks it is. The socket was from what I can tell in ESTABLISHED.

I'm completely out of ideas how to continue debugging this especially because it's impossible for me to reproduce on anything other than this particular amazon machine. It does not appear on OS X, on two linux VMs running in virtualbox, not happening from one of my linux servers on hetzner to another one there and I could also not reproduce it form hetzner to amazon though in that particular case I saw some other artifacts that might have prevented debugging this (in particular the performance was quite bad and it continued on for a day without incidents).

I first thought this mighe be #1845 but from what I can tell there is no high CPU usage involved. It just blocks on a read. If I can help in any way with investigating this further, please let me know.