Issue 1:
Checklist

I have verified that the issue exists against the master branch of Celery.

This has already been asked to the discussion group first.

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

None

Possible Duplicates

None

Environment & Settings

Celery version: 4.3.0 (rhubarb)

celery report Output:

software -> celery:4.3.0 (rhubarb) kombu:4.6.11 py:3.6.12
billiard:3.6.3.0 redis:3.3.11
platform -> system:Linux arch:64bit, ELF
kernel version:4.14.106-97.85.amzn2.x86_64 imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:redis results:redis://query-runner-redis-master:6379/0

BROKER_URL: 'redis://query-runner-redis-master:6379/0'
CELERY_RESULT_BACKEND: 'redis://query-runner-redis-master:6379/0'
CELERY_INCLUDE: ['tasks.query']
CELERY_TIMEZONE: 'UTC'
CELERY_TASK_SERIALIZER: 'json'
CELERY_IGNORE_RESULT: True
CELERY_TASK_ACKS_LATE: True
CELERYD_PREFETCH_MULTIPLIER: 1
CELERY_ROUTES: {
'tasks.query.kill_query': {'queue': 'meta'},
'tasks.query.run_query': {'queue': 'default'}}
CELERY_ALWAYS_EAGER: False
CELERYBEAT_SCHEDULE: {
}
CELERY_TASK_CREATE_MISSING_QUEUES: True

Steps to Reproduce

Required Dependencies

Minimal Python Version: N/A or Unknown

Minimal Celery Version: N/A or Unknown

Minimal Kombu Version: N/A or Unknown

Minimal Broker Version: N/A or Unknown

Minimal Result Backend Version: N/A or Unknown

Minimal OS and/or Kernel Version: N/A or Unknown

Minimal Broker Client Version: N/A or Unknown

Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

amqp==2.6.1
billiard==3.6.3.0
blinker==1.4
boto==2.49.0
boto3==1.16.21
botocore==1.19.21
celery==4.3.0
celery-prometheus-exporter==1.7.0
certifi==2020.11.8
chardet==3.0.4
cheroot==8.4.7
CherryPy==18.6.0
click==7.1.2
datadog==0.39.0
decorator==4.4.2
env-excavator==1.5.0
Flask==1.1.2
Flask-Cors==3.0.9
flask-swagger==0.2.14
flask-swagger-ui==3.36.0
flower==0.9.5
future==0.18.2
gunicorn==20.0.4
humanize==3.1.0
idna==2.10
importlib-metadata==2.0.0
importlib-resources==3.3.0
inflection==0.5.1
influxdb==5.3.1
itsdangerous==1.1.0
jaraco.classes==3.1.0
jaraco.collections==3.0.0
jaraco.functools==3.0.1
jaraco.text==3.2.0
JayDeBeApi==1.1.1
Jinja2==2.11.2
jmespath==0.10.0
JPype1==0.6.3
kombu==4.6.11
MarkupSafe==1.1.1
more-itertools==8.6.0
msgpack==1.0.0
numpy==1.17.0
portend==2.7.0
prometheus-client==0.9.0
prometheus-flask-exporter==0.18.1
psycopg2==2.8.6
pycrypto==2.6.1
PyHive==0.6.3
PyMySQL==0.10.1
python-dateutil==2.8.1
pytz==2020.4
PyYAML==5.3.1
qds-sdk==1.16.1
raven==6.10.0
redis==3.3.11
redlock==1.2.0
requests==2.25.0
s3transfer==0.3.3
six==1.15.0
tempora==4.0.1
thrift==0.13.0
tornado==6.1
urllib3==1.26.2
vine==1.3.0
Werkzeug==1.0.1
zc.lockfile==2.0
zipp==3.4.0

Other Dependencies

N/A

Minimally Reproducible Test Case

Expected Behavior

I was hoping by specifying -Ofair that tasks would not be reserved by workers as stated here https://docs.celeryproject.org/en/v4.3.0/userguide/optimizing.html?highlight=optimization#prefork-pool-prefetch-settings

Actual Behavior

Workers have reserved tasks while other workers are free to do work.

I am going to paste the state we are seeing in prod when a lot of tasks come in. Notice how query-runner-celery-datalake-interval-refresh-784bb858c8-qskvj has a reserved task while celery@query-runner-celery-datalake-interval-refresh-784bb858c8-dgmf9 is empty and could run that task. Please note the name of the worker is the name of the queue they are reading from.

celery inspect active and celery inspect reserved Output:

root@query-runner-celery-datalake-687847cbb6-mxrnc:/opt/workdir# celery -A tasks inspect active
2020-11-24 21:03:59,312 [WARNING] datadog.api: No agent or invalid configuration file found
2020-11-24 21:03:59,316 [INFO] tasks.datadog_helper: initialized with
2020-11-24 21:03:59,316 [INFO] tasks.datadog_helper: {'statsd_host': '172.29.156.91', 'statsd_port': 8125, 'statsd_namespace': 'query_runner_celery_tasks'}
-> celery@query-runner-celery-default-6c9bd65fd7-4cdz6: OK
- empty -
-> celery@query-runner-celery-default-6c9bd65fd7-v8w22: OK
- empty -
-> celery@query-runner-celery-datalake-687847cbb6-68xqn: OK
- empty -
-> celery@query-runner-celery-ga-55cdb77d4-lsk2z: OK
- empty -
-> celery@query-runner-celery-datalake-687847cbb6-mxrnc: OK
- empty -
-> celery@query-runner-celery-default-6c9bd65fd7-9f8r2: OK
- empty -
-> celery@query-runner-celery-datalake-687847cbb6-hnrhw: OK
- empty -
-> celery@query-runner-celery-ga-55cdb77d4-9jp2n: OK
- empty -
-> celery@query-runner-celery-datalake-687847cbb6-4dx2b: OK
- empty -
-> celery@query-runner-celery-datalake-687847cbb6-q82hx: OK
- empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-nqhtk: OK
* {'id': '93a32a04-7a6d-4a32-a2ee-8b88ca4f9049', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_aa42f1c1970bc1ac2778a65c60b2e08b', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3107', 'caller_json': {'queryId': 3107, 'interval': 'HOURLY', 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-nqhtk', 'time_start': 1606251603.8548772, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 15}
* {'id': '747a1dd7-209a-4870-a636-c18176587723', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_373fafa1388d047c83cae9d40f249655', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3149 on dash 616', 'caller_json': {'queryId': 3149, 'interval': 'HOURLY', 'dashboard_id': 616, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-nqhtk', 'time_start': 1606251609.6809852, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 13}
* {'id': '28ec904f-197f-4aef-99e2-0887f6f85ec7', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_829e5787764013fbf5eb0958eec86a8f', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2400 on dash 609', 'caller_json': {'queryId': 2400, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-nqhtk', 'time_start': 1606251604.9346218, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 16}
* {'id': '705d4f46-662d-4930-9011-265e1065346d', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_04cd73d13695b2a7d9503148018c2650', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3179 on dash 616', 'caller_json': {'queryId': 3179, 'interval': 'HOURLY', 'dashboard_id': 616, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-nqhtk', 'time_start': 1606240820.285725, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 14}
-> celery@query-runner-celery-default-6c9bd65fd7-4s4lz: OK
- empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-92hxc: OK
* {'id': '7afd7a99-368e-4995-8e5a-a1802dff7476', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_9fe008b693e6dc12999bf58212430afa', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2403 on dash 609', 'caller_json': {'queryId': 2403, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-92hxc', 'time_start': 1606251605.1041203, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 15}
* {'id': '4d59fc3f-40b9-4d51-ad41-e241518c629f', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_9a6f5ab9f25ab21bddcc128aee8677d6', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2396 on dash 609', 'caller_json': {'queryId': 2396, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-92hxc', 'time_start': 1606251604.2693179, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 16}
* {'id': '05bc6789-2183-47d2-abe0-7093bad200a5', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_a937a6def4c1010c15dd3c5e1b488d2d', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3116 on dash 616', 'caller_json': {'queryId': 3116, 'interval': 'HOURLY', 'dashboard_id': 616, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-92hxc', 'time_start': 1606251609.2829497, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 14}
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-269rc: OK
* {'id': 'd6ddf309-e58a-4f2d-be71-1824e618716e', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_b7253439941188da3776ffdc8bab2b07', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2401 on dash 609', 'caller_json': {'queryId': 2401, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-269rc', 'time_start': 1606251605.0180476, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 16}
* {'id': '62bb08c5-d6a6-4966-8de1-0d872faba09f', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_b75d10ba197e1bb1ab787217ca5daa96', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2395 on dash 609', 'caller_json': {'queryId': 2395, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-269rc', 'time_start': 1606251604.1448753, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 15}
* {'id': '82240c16-ec84-4279-9cd4-694a73a145eb', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_6435b6004863fd5a10de1abc8bf294cd', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3188 on dash 624', 'caller_json': {'queryId': 3188, 'interval': 'HOURLY', 'dashboard_id': 624, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-269rc', 'time_start': 1606251609.9815636, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 14}
* {'id': '9c062ae6-bc8d-44d1-b861-068cd1032c38', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_6912fef8aee81d8aa0a68f1326e5bbf6', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3117 on dash 616', 'caller_json': {'queryId': 3117, 'interval': 'HOURLY', 'dashboard_id': 616, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-269rc', 'time_start': 1606251609.2058136, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 13}
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-94qs4: OK
* {'id': '87228aca-e02f-48a3-9096-5540c7b4fb22', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_e97a5e28903b3652a4a2c386fc287cc6', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3115 on dash 616', 'caller_json': {'queryId': 3115, 'interval': 'HOURLY', 'dashboard_id': 616, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-94qs4', 'time_start': 1606251609.4260392, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 13}
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-6xldj: OK
* {'id': '1a8982d5-32cd-4fcf-a134-72a79d7e5da7', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_02758ef185f77a7397ead9e8e90e9743', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2397 on dash 609', 'caller_json': {'queryId': 2397, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-6xldj', 'time_start': 1606251604.4064894, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 16}
* {'id': 'b2804139-6d51-4ffe-817b-8b9efc7a9cd4', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_1920e1f67f84ac24fb3d2b8e8bb4bd22', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3114 on dash 616', 'caller_json': {'queryId': 3114, 'interval': 'HOURLY', 'dashboard_id': 616, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-6xldj', 'time_start': 1606251609.3467886, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 15}
-> celery@query-runner-celery-ga-55cdb77d4-j6994: OK
- empty -
-> celery@query-runner-celery-datalake-687847cbb6-m8wfp: OK
- empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-dgmf9: OK
- empty -
-> celery@query-runner-celery-datalake-687847cbb6-drdl2: OK
- empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-lgstj: OK
* {'id': '741f2751-ca24-4bcd-b78c-1d92e679952d', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_e526e7f0e7a7c28271dc8f08ee19504b', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3021 on dash 595', 'caller_json': {'queryId': 3021, 'interval': 'HOURLY', 'dashboard_id': 595, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-lgstj', 'time_start': 1606251616.69369, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 16}
-> celery@query-runner-celery-ga-55cdb77d4-6jxzz: OK
- empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-g9swh: OK
* {'id': '650009cc-4d6b-42c4-96b1-9a3112a1e5c8', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_711f6cdfe13c22710887f00f5678baeb', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2399 on dash 609', 'caller_json': {'queryId': 2399, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-g9swh', 'time_start': 1606251604.6513581, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 15}
* {'id': 'd74e3272-45f0-44ee-bcab-7b67de60b407', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_5ca350ca8b4b185ad42f94c6dd3b6fc9', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3189 on dash 624', 'caller_json': {'queryId': 3189, 'interval': 'HOURLY', 'dashboard_id': 624, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-g9swh', 'time_start': 1606251611.4904687, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 14}
* {'id': '6f61b237-fa9e-4b30-8131-8d8ee75306c5', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_9bc92aabee38e8d6d607bbb24785d814', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2137 on dash 557', 'caller_json': {'queryId': 2137, 'interval': 'HOURLY', 'dashboard_id': 557, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-g9swh', 'time_start': 1606248030.1126661, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 13}
-> celery@query-runner-celery-datalake-687847cbb6-dvdpl: OK
- empty -
-> celery@query-runner-celery-datalake-687847cbb6-pdxsk: OK
- empty -
-> celery@query-runner-celery-ga-55cdb77d4-vl2t6: OK
- empty -
-> celery@query-runner-celery-default-6c9bd65fd7-g55gl: OK
- empty -
-> celery@query-runner-celery-datalake-687847cbb6-7spd4: OK
- empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-qskvj: OK
* {'id': 'ab45784f-8d85-452c-b690-43ead1a010c6', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_bb8a680bab37f6bc86eb9f5c7435d656', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2816 on dash 609', 'caller_json': {'queryId': 2816, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-qskvj', 'time_start': 1606251605.576281, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 14}
* {'id': '4e64157e-9356-4362-88af-318013c0c4de', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_4859064fc4b96df99eec68e94f733157', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3143 on dash 616', 'caller_json': {'queryId': 3143, 'interval': 'HOURLY', 'dashboard_id': 616, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-qskvj', 'time_start': 1606251609.5426545, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 13}
* {'id': 'ac0c403b-ade6-4377-9a41-6d305268204b', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_b04a82ce9376c9c6618c3a300abe4182', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3109', 'caller_json': {'queryId': 3109, 'interval': 'HOURLY', 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-qskvj', 'time_start': 1606240815.8285584, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 16}
* {'id': 'e9c368de-d47f-4a84-b05d-81cbda00a394', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_4d03e965dbd57a05d6071c0b5cbf699d', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2398 on dash 609', 'caller_json': {'queryId': 2398, 'interval': 'HOURLY', 'dashboard_id': 609, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-qskvj', 'time_start': 1606251604.517987, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 15}
-> celery@query-runner-celery-default-6c9bd65fd7-8zdvm: OK
- empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-2zxhc: OK
* {'id': '03e786df-44b1-412a-8354-3acbc762528f', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_0687aa16794011f4d1a0ee1023173bf8', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3105 on dash 557', 'caller_json': {'queryId': 3105, 'interval': 'HOURLY', 'dashboard_id': 557, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-2zxhc', 'time_start': 1606251618.9102547, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 16}
* {'id': '646617ce-9835-4b60-b5fd-c4a4a5631749', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_ed09cf8c1166115a4ed1a5aa33b032c7', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 3144 on dash 616', 'caller_json': {'queryId': 3144, 'interval': 'HOURLY', 'dashboard_id': 616, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-2zxhc', 'time_start': 1606251609.6082683, 'acknowledged': True, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': 15}
root@query-runner-celery-datalake-687847cbb6-mxrnc:/opt/workdir# celery -A tasks inspect reserved
2020-11-24 21:04:03,851 [WARNING] datadog.api: No agent or invalid configuration file found
2020-11-24 21:04:03,854 [INFO] tasks.datadog_helper: initialized with
2020-11-24 21:04:03,855 [INFO] tasks.datadog_helper: {'statsd_host': '172.29.156.91', 'statsd_port': 8125, 'statsd_namespace': 'query_runner_celery_tasks'}
-> celery@query-runner-celery-default-6c9bd65fd7-4cdz6: OK
- empty -
-> celery@query-runner-celery-default-6c9bd65fd7-v8w22: OK
- empty -
-> celery@query-runner-celery-datalake-687847cbb6-mxrnc: OK
- empty -
-> celery@query-runner-celery-datalake-687847cbb6-68xqn: OK
- empty -
-> celery@query-runner-celery-datalake-687847cbb6-q82hx: OK
- empty -
-> celery@query-runner-celery-ga-55cdb77d4-lsk2z: OK
- empty -
-> celery@query-runner-celery-datalake-687847cbb6-4dx2b: OK
- empty -
-> celery@query-runner-celery-datalake-687847cbb6-hnrhw: OK
- empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-lgstj: OK
- empty -
-> celery@query-runner-celery-default-6c9bd65fd7-9f8r2: OK
- empty -
-> celery@query-runner-celery-ga-55cdb77d4-j6994: OK
- empty -
-> celery@query-runner-celery-ga-55cdb77d4-9jp2n: OK
- empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-92hxc: OK
- empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-269rc: OK
- empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-nqhtk: OK
- empty -
-> celery@query-runner-celery-default-6c9bd65fd7-4s4lz: OK
- empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-94qs4: OK
- empty -
-> celery@query-runner-celery-datalake-687847cbb6-m8wfp: OK
- empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-6xldj: OK
- empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-dgmf9: OK
- empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-g9swh: OK
- empty -
-> celery@query-runner-celery-ga-55cdb77d4-6jxzz: OK
- empty -
-> celery@query-runner-celery-datalake-687847cbb6-dvdpl: OK
- empty -
-> celery@query-runner-celery-default-6c9bd65fd7-g55gl: OK
- empty -
-> celery@query-runner-celery-datalake-687847cbb6-drdl2: OK
- empty -
-> celery@query-runner-celery-ga-55cdb77d4-vl2t6: OK
- empty -
-> celery@query-runner-celery-default-6c9bd65fd7-8zdvm: OK
- empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-qskvj: OK
* {'id': '3f501bbe-6c56-48e2-ba64-9b1adac30695', 'name': 'tasks.query.run_query', 'args': '()', 'kwargs': "{'query_id': 'query_6eb04958c42a71ebd014f9be9e116f73', 'queue': 'datalake-interval-refresh', 'caller_info': {'caller': 'Interval refresh HOURLY for Query id 2823 on dash 557', 'caller_json': {'queryId': 2823, 'interval': 'HOURLY', 'dashboard_id': 557, 'action': 'refresh_data'}, 'user_id': '0', 'username': 'intervalRefreshUser'}}", 'type': 'tasks.query.run_query', 'hostname': 'celery@query-runner-celery-datalake-interval-refresh-784bb858c8-qskvj', 'time_start': None, 'acknowledged': False, 'delivery_info': {'exchange': '', 'routing_key': 'datalake-interval-refresh', 'priority': 0, 'redelivered': None}, 'worker_pid': None}
-> celery@query-runner-celery-datalake-687847cbb6-7spd4: OK
- empty -
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-2zxhc: OK
- empty -
-> celery@query-runner-celery-datalake-687847cbb6-pdxsk: OK
- empty -

Also, here is the output of celery inspect stats where you can see fair is getting in properly

celery inspect stats for worker query-runner-celery-datalake-interval-refresh-784bb858c8-dgmf9 Output:

```
-> celery@query-runner-celery-datalake-interval-refresh-784bb858c8-dgmf9: OK
{
"broker": {
"alternates": [],
"connect_timeout": 4,
"failover_strategy": "round-robin",
"heartbeat": 120.0,
"hostname": "query-runner-redis-master",
"insist": false,
"login_method": null,
"port": 6379,
"ssl": false,
"transport": "redis",
"transport_options": {},
"uri_prefix": null,
"userid": null,
"virtual_host": "0"
},
"clock": "8600271",
"pid": 8,
"pool": {
"max-concurrency": 4,
"max-tasks-per-child": "N/A",
"processes": [
13,
14,
15,
16
],
"put-guarded-by-semaphore": false,
"timeouts": [
0,
0
],
"writes": {
"all": "42.86%, 42.86%, 14.29%",
"avg": "33.33%",
"inqueues": {
"active": 0,
"total": 4
},
"raw": "3, 3, 1",
"strategy": "fair",
"total": 7
}
},
"prefetch_count": 4,
"rusage": {
"idrss": 0,
"inblock": 72400,
"isrss": 0,
"ixrss": 0,
"majflt": 158,
"maxrss": 68800,
"minflt": 69557,
"msgrcv": 0,
"msgsnd": 0,
"nivcsw": 2942,
"nsignals": 0,
"nswap": 0,
"nvcsw": 334278,
"oublock": 1584,
"stime": 7.775837,
"utime": 152.889586
},
"total": {
"tasks.query.run_query": 7
}
}
```

Generally, we have a lot of tasks long running tasks. This issue is making our tasks take double the amount of time because they are unnecessarily waiting behind other tasks without utilizing the other workers.

Issue 2:
Checklist

[x ] I have verified that the issue exists against the master branch of Celery.

[ x] This has already been asked to the discussion group first.

[ x] I have read the relevant section in the

contribution guide

on reporting bugs.

[ x] I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

[x ] I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

[ x] I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

None

Possible Duplicates

#5748

Environment & Settings

Celery version: 4.4.0

celery report Output:

Steps to Reproduce

I first start Celery with Beat option (-B).

It add_periodic_task after config as specified in the doc (https://docs.celeryproject.org/en/stable/userguide/periodic-tasks.html#starting-the-scheduler)

Then, I try to add or remove a task though API Flask app, by either:

modifying the app.conf.beat_schedule (remove or add task by name

modifying the celerybeat-schedule shelve

Required Dependencies

Minimal Python Version: 3.7.4

Minimal Celery Version: 4.4.°

Minimal Kombu Version: N/A or Unknown

Minimal Broker Version: RabboitMQ 3.8.0

Minimal Result Backend Version: N/A or Unknown

Minimal OS and/or Kernel Version: Redhat 7

Minimal Broker Client Version: N/A or Unknown

Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

amqp==2.5.2
astroid==2.3.3
attrs==19.3.0
autopep8==1.4.4
Babel==2.7.0
billiard==3.6.3.0
celery==4.4.0
certifi==2019.9.11
chardet==3.0.4
Click==7.0
coverage==4.5.4
fixtures-mongoengine==1.3.1
Flask==1.1.1
Flask-Babel==0.12.2
Flask-Session==0.3.1
Flask-WTF==0.14.2
idna==2.8
importlib-metadata==1.5.0
isodate==0.6.0
isort==4.3.21
itsdangerous==1.1.0
Jinja2==2.11.1
jsonschema==3.2.0
kombu==4.6.8
lazy-object-proxy==1.4.3
MarkupSafe==1.1.1
mccabe==0.6.1
mongoengine==0.18.2
paho-mqtt==1.4.0
pycodestyle==2.5.0
pylint==2.4.2
pymongo==3.9.0
pyrsistent==0.15.7
pytz==2019.3
PyYAML==5.3
requests==2.22.0
semver==2.9.0
six==1.12.0
typed-ast==1.4.1
urllib3==1.25.6
vine==1.3.0
Werkzeug==0.16.0
wrapt==1.11.2
WTForms==2.2.1
zipp==2.1.0

Other Dependencies

N/A

Minimally Reproducible Test Case

Expected Behavior

I would like to be able to add or remove a task dynmically.

Actual Behavior

In fact nothing happen, whatever I do, If I remove the task either from shelve (as suggested here: https://stackoverflow.com/questions/10194975/how-to-dynamically-add-remove-periodic-tasks-to-celery-celerybeat proposed by Tristant Brown) or from config app.conf.beat_schedule, the task continue to be schedule at the original cron rules I provided.

I first though it was an issue in the sync processed which happend every 3 minutes, and after looking at the code, I discovered that in beat.py file line 421, the function sync does nothing..