Issue 1:
Tested with latest pypi version (4.0.2) and against master branch.

Upgraded recently from 3.1.25 as I had some issues with tasks being received but executed after a short delay (as in #2771). Now it got worse.

Tried using librabbitmq, but encountered: #3675 when I was queueing task inside a task.

Steps to reproduce

Reproduced it with example project from: https://github.com/celery/celery/blob/master/examples/app/myapp.py

Installed rabbitmq locally ({rabbit,"RabbitMQ","3.6.9"})

Ran celery worker -A myapp -l debug

In another process, sent task a few times: app.send_task('myapp.add', args=(1,2))

Expected behavior

Tasks should be processed asap.

Actual behavior

Sometimes there is a pretty big delay between receiving and processing a task.

[2017-04-14 02:18:08,889: INFO/MainProcess] Received task: myapp.add[59816272-ca82-4eee-ba95-8d71da5dca52]
[2017-04-14 02:18:08,889: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x1064d01b8> (args:('myapp.add', '59816272-ca82-4eee-ba95-8d71da5dca52', {'origin': 'gen84425@RK-MacBook.local', 'lang': 'py', 'task': 'myapp.add', 'group': None, 'root_id': '59816272-ca82-4eee-ba95-8d71da5dca52', u'delivery_info': {u'priority': 0, u'redelivered': False, u'routing_key': u'celery', u'exchange': u''}, 'expires': None, u'correlation_id': '59816272-ca82-4eee-ba95-8d71da5dca52', 'retries': 0, 'timelimit': [None, None], 'argsrepr': '(1, 2)', 'eta': None, 'parent_id': None, u'reply_to': '51344e5c-09df-31a0-bbfd-e64295f6db67', 'id': '59816272-ca82-4eee-ba95-8d71da5dca52', 'kwargsrepr': '{}'}, u'[[1, 2], {}, {"chord": null, "callbacks": null, "errbacks": null, "chain": null}]', 'application/json', 'utf-8') kwargs:{})
[2017-04-14 02:18:10,746: DEBUG/MainProcess] Task accepted: myapp.add[59816272-ca82-4eee-ba95-8d71da5dca52] pid:84171
[2017-04-14 02:18:10,747: INFO/PoolWorker-8] Task myapp.add[59816272-ca82-4eee-ba95-8d71da5dca52] succeeded in 0.000706910970621s: 3

Above, between TaskPool: Apply and Task accepted: there is a 2s delay. Usually it's smaller, like 0.5s. But it's very inconsistent.

Tried with redis broker - no delays and overall much faster. But I am hesitant to make the jump on production and I don't believe this behavior is to be expected anyway.

Tested librabbitmq - looks good, also no delays. But I am getting #3675 (which is closed for some reason).

Issue 2:
I have a service using Celery 4.0.2 to dispatch tasks to workers. I observed that, periodically, Celery workers hang altogether for a while without doing any tasks, and then suddenly, they revive and resume execution again. Looks like Celery 4.0.2 buffers those tasks and release to workers later.

Celery 3 (I tried 3.1.23 and 3.1.25) don't have those problems and tasks are processed smoothly & evenly.

I use Python Celery module. The command is something like "-A module_name --loglevel=info -n str(uuid4())[:8] -P solo --autoscale=16,2";

Some other configs are like:

BROKER_CONNECTION_TIMEOUT=3600 * 24,

BROKER_CONNECTION_MAX_RETRIES=None,

CELERY_ACCEPT_CONTENT=['application/json'],

CELERY_TASK_SERIALIZER='json',

CELERY_RESULT_SERIALIZER='json',

CELERY_ACKS_LATE=False

Some screenshots are as follows;

Celery 4

Celery 3.0.25