Issue 1:
Checklist

I have verified that the issue exists against the master branch of Celery.

This has already been asked to the discussion group first.

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

[ x ] I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

celery/kombu#1102

Possible Duplicates

celery/kombu#1102

Environment & Settings

Celery version:

5.0.0rc3 (singularity)

celery report Output:

[ec2-user@ip-172-31-50-49 ~]$ celery -A celery_worker report

software -> celery:5.0.0rc3 (singularity) kombu:5.0.2 py:3.7.9
billiard:3.6.3.0 sqs:N/A
platform -> system:Linux arch:64bit, ELF
kernel version:4.14.193-149.317.amzn2.x86_64 imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:sqs results:disabled

broker_url: 'sqs://localhost//'

Steps to Reproduce

Required Dependencies

Minimal Python Version: N/A or Unknown

Minimal Celery Version: 4.4.3

Minimal Kombu Version: N/A or Unknown

Minimal Broker Version: N/A or Unknown

Minimal Result Backend Version: N/A or Unknown

Minimal OS and/or Kernel Version: N/A or Unknown

Minimal Broker Client Version: N/A or Unknown

Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

amqp==5.0.1
awscli==1.18.139
backcall==0.2.0
billiard==3.6.3.0
boto3==1.14.62
botocore==1.17.62
celery==5.0.0rc3
click==7.1.2
click-didyoumean==0.0.3
click-repl==0.1.6
colorama==0.4.3
decorator==4.4.2
docutils==0.15.2
importlib-metadata==1.7.0
ipython==7.18.1
ipython-genutils==0.2.0
jedi==0.17.2
jmespath==0.10.0
kombu==5.0.2
parso==0.7.1
pexpect==4.8.0
pickleshare==0.7.5
prompt-toolkit==3.0.7
ptyprocess==0.6.0
pyasn1==0.4.8
pycurl==7.43.0.6
Pygments==2.7.0
python-dateutil==2.8.1
pytz==2020.1
PyYAML==5.3.1
rsa==4.5
s3transfer==0.3.3
six==1.15.0
traitlets==5.0.4
urllib3==1.25.10
vine==5.0.0
wcwidth==0.2.5
zipp==3.1.0

Other Dependencies

N/A

Minimally Reproducible Test Case

Schedule 1000 simple 4+4 tasks from howto guide, launch worker.

In [33]: %time for x in range(1000):add.apply_async(args=(4, 4))
CPU times: user 3.4 s, sys: 61.7 ms, total: 3.46 s
Wall time: 12.7 s

from celery import Celery

app = Celery('tasks', broker='sqs://')

@app.task
def add(x, y):
return x + y

Expected Behavior

Higher throughput with higher CPU utilisation

Actual Behavior

Worker processes ~10 messages per second. CPU utilisation is low, looks like celery waits most of the time.

Celery and all test code runs was tested on fresh EC2 instance in us-east-1, with SQS in the same region.

At the same time following code allows to consume all 1000 messages from queue in ~10 seconds, event without using bulk deletes or any async APIs:

In [30]: %time for x in range(1000):add.apply_async(args=(4, 4))
CPU times: user 3.44 s, sys: 80.7 ms, total: 3.52 s
Wall time: 12.9 s

In [31]: def get_all():
...: while True:
...: messages = client.receive_message(QueueUrl='https://queue.amazonaws.com/385755224306/celery', MaxNumberOfMessages=10)
...: if 'Messages' not in messages:
...: break
...: for message in messages['Messages']:
...: client.delete_message(QueueUrl='https://queue.amazonaws.com/385755224306/celery', ReceiptHandle=message['ReceiptHandle'])
...:
In [35]: %time get_all()
CPU times: user 2.9 s, sys: 84.9 ms, total: 2.98 s
Wall time: 9.72 s

There are visible delays in celery worker log output, for example:

[2020-09-16 14:02:14,573: INFO/ForkPoolWorker-1] Task celery_worker.add[88d2fce8-0d0b-49fe-b0a7-2713b8ab6248] succeeded in 4.8579999202047475e-05s: 8
[2020-09-16 14:02:15,489: INFO/MainProcess] Received task: celery_worker.add[76ec9b77-bd42-4042-9933-35554e678506]

Looks like delay always happens once tasks have finished execution, but before new tasks were received from queue.

Tests and timings:

-c1, default prefetch -> 9m30s

-c5, default prefetch -> 3m30s

-c10, default prefetch -> 1m45s

-c20, default prefetch -> 1m44s

-c1, prefetch 100 -> 1m45s

Looks like celery throughput somehow gets limited by SQS MaxNumberOfMessages=10 limitation, despite concurrency or prefetch settings. However, boto3 sample script shows that AWS API allows to consume messages match faster, so there should be some possible optimisation for Celery/Kombu.

Issue 2:
Checklist

I have verified that the issue exists against the master branch of Celery.

This has already been asked to the discussions forum first.

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

None

Possible Duplicates

None

Environment & Settings

Celery version: 5.2.1 (dawn-chorus)

celery report Output:

Traceback (most recent call last):
File "/tmp/test/venv/bin/celery", line 8, in <module>
sys.exit(main())
File "/tmp/test/venv/lib/python3.9/site-packages/celery/__main__.py", line 15, in main
sys.exit(_main())
File "/tmp/test/venv/lib/python3.9/site-packages/celery/bin/celery.py", line 213, in main
return celery(auto_envvar_prefix="CELERY")
File "/tmp/test/venv/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/tmp/test/venv/lib/python3.9/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/tmp/test/venv/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/tmp/test/venv/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/tmp/test/venv/lib/python3.9/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/tmp/test/venv/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
TypeError: report() got an unexpected keyword argument 'ini'

Steps to Reproduce

Define customer user option and run celery -A proj multi start --my-custom-option=foo.

Required Dependencies

Minimal Python Version: N/A or Unknown

Minimal Celery Version: N/A or Unknown

Minimal Kombu Version: N/A or Unknown

Minimal Broker Version: N/A or Unknown

Minimal Result Backend Version: N/A or Unknown

Minimal OS and/or Kernel Version: N/A or Unknown

Minimal Broker Client Version: N/A or Unknown

Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

Other Dependencies

N/A

Minimally Reproducible Test Case

Expected Behavior

Celery multi should work with user options like celery worker

Actual Behavior

Starting an application that has custom options with celery multi start fails.

# celery -A proj multi start worker1 \
--ini=config.ini \
--concurrency=4 -Q celery
/tmp/test/venv/lib/python3.9/site-packages/pkg_resources/__init__.py:116: PkgResourcesDeprecationWarning: 1.14.0-unknown is an invalid version and will not be supported in a future release
warnings.warn(
Traceback (most recent call last):
File "/tmp/test/venv/bin/celery", line 8, in <module>
sys.exit(main())
File "/tmp/test/venv/lib/python3.9/site-packages/celery/__main__.py", line 15, in main
sys.exit(_main())
File "/tmp/test/venv/lib/python3.9/site-packages/celery/bin/celery.py", line 213, in main
return celery(auto_envvar_prefix="CELERY")
File "/tmp/test/venv/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/tmp/test/venv/lib/python3.9/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/tmp/test/venv/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/tmp/test/venv/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/tmp/test/venv/lib/python3.9/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/tmp/test/venv/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/tmp/test/venv/lib/python3.9/site-packages/celery/bin/base.py", line 134, in caller
return f(ctx, *args, **kwargs)
TypeError: multi() got an unexpected keyword argument 'ini'

The multi function in celery/celery/bin/multi.py only accepts the ctx parameter, but

the handle_preload_options decorator is calling f(ctx, *args, **kwargs).