Issue 1:
Checklist

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

None

Possible Duplicates

None

Environment & Settings

Celery version: 4.3.0

celery report Output:

Steps to Reproduce

Required Dependencies

Minimal Python Version: 3.7.1

Minimal Celery Version: 4.3.0

Minimal Kombu Version: 4.2.1

Minimal Broker Version: redis==2.10.6

Minimal Result Backend Version: N/A or Unknown

Minimal OS and/or Kernel Version: N/A or Unknown

Minimal Broker Client Version: redis==2.10.6

Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

amqp==2.2.2
appdirs==1.4.3
appnope==0.1.0
argh==0.26.2
attrs==17.4.0
autoflake==1.1
Babel==2.6.0
backports.shutil-get-terminal-size==1.0.0
billiard==3.5.0.3
bleach==3.0.2
boto3==1.7.22
botocore==1.10.22
cachetools==3.1.0
celery==4.3.0
certifi==2018.1.18
chardet==3.0.4
colorama==0.3.9
contextlib2==0.5.5
coverage==4.5.1
decorator==4.2.1
defusedxml==0.5.0
dj-database-url==0.5.0
Django==2.2
django-admin-rangefilter==0.3.12
django-allauth==0.35.0
django-celery-beat==1.1.1
django-cors-middleware==1.3.1
django-cursor-pagination==0.1.2
django-extensions==2.0.0
django-filter==1.1.0
django-formtools==2.1
django-otp==0.5.0
django-phonenumber-field==1.3.0
django-prometheus==1.0.15
django-two-factor-auth==1.8.0
django-viewflow-pro==1.4.2
djangorestframework==3.7.7
djangorestframework-simplejwt==3.2.2
djproxy==2.3.4
docopt==0.6.2
docutils==0.14
elasticsearch==6.0.0
enum34==1.1.6
ephem==3.7.6.0
factory-boy==2.11.1
Faker==0.8.11
flake8==3.5.0
flake8-isort==2.4
flake8-polyfill==1.0.2
freezegun==0.3.11
gevent==1.3.7
google-api-core==1.7.0
google-auth==1.6.2
googleapis-common-protos==1.5.6
graphene==2.0.1
graphene-django==2.0.0
graphql-core==2.0
graphql-relay==0.4.5
greenlet==0.4.15
grpcio==1.17.1
grpcio-tools==1.17.1
gunicorn==19.9.0
idna==2.6
img2pdf==0.3.2
ipdb==0.11
ipython==6.2.1
ipython-genutils==0.2.0
iso8601==0.1.12
isort==4.3.4
jedi==0.11.1
jmespath==0.9.3
jsonfield==2.0.2
kombu==4.2.1
mccabe==0.6.1
mock==2.0.0
mypy==0.641
mypy-extensions==0.4.1
mypy-protobuf==1.7
numpy==1.16.2
oauthlib==2.0.6
opencensus==0.1.10
packaging==17.1
parso==0.1.1
pathlib2==2.3.0
pathtools==0.1.2
pbr==3.1.1
pdfrw==0.4
permissive-search==0.1.7
pexpect==4.4.0
phonenumberslite==8.10.4
pickleshare==0.7.4
Pillow==5.4.1
pkginfo==1.4.2
pluggy==0.6.0
prometheus-client==0.5.0
promise==2.1
prompt-toolkit==1.0.15
protobuf==3.6.1
psycopg2-binary==2.7.6.1
ptyprocess==0.5.2
py==1.5.2
pyasn1==0.4.5
pyasn1-modules==0.2.4
pycodestyle==2.3.1
pycountry==18.5.26
pycurl==7.43.0.1
pyflakes==1.6.0
Pygments==2.2.0
PyJWT==1.6.1
pyparsing==2.2.0
pytest==3.4.1
pytest-cov==2.5.1
pytest-django==3.1.2
pytest-watch==4.1.0
pythia-python==1.5.2
python-dateutil==2.7.0
python-http-client==3.1.0
python3-openid==3.1.0
pytz==2018.3
PyYAML==3.12
qrcode==6.1
raven==6.6.0
readme-renderer==24.0
redis==2.10.6
reportlab==3.5.13
requests==2.20.0
requests_aws4auth==0.9
requests-mock==1.4.0
requests-oauthlib==0.8.0
requests-toolbelt==0.8.0
rsa==4.0
Rx==1.6.1
s3transfer==0.1.13
scandir==1.7
sendgrid==5.3.0
simplegeneric==0.8.1
singledispatch==3.4.0.3
six==1.11.0
snapshottest==0.5.0
termcolor==1.1.0
testfixtures==5.4.0
text-unidecode==1.2
tqdm==4.19.8
traitlets==4.3.2
twine==1.12.1
typed-ast==1.1.0
typing==3.6.4
ulid==1.1
Unidecode==1.0.22
urllib3==1.22
vine==1.1.4
watchdog==0.8.3
wcwidth==0.1.7
webencodings==0.5.1
whitenoise==3.3.1
wrapt==1.10.11
yapf==0.20.2

Other Dependencies

N/A

Minimally Reproducible Test Case

@celery_app.task(name="some_name")
def some_function(
*,
first_kw: str = "default value 1",
second_kw: str,
third_kw: str,
n_minus_one_kw: str,
last_kw: str = "default value 2"
):
service.make_service(
kw_1=first_kw,
kw_2=second_lw,
kw_3=third_kw,
kw_4=n_minus_one_kw,
kw_5=last_kw,
)

Expected Behavior

I expect that my function kwargs with default values will be optional. In example I suppose that first_kw and last_kw will be optional, and other will be required keyword arguments.

Actual Behavior

kwonlyargs_optional = [(kw, i) for i, kw in enumerate(spec.kwonlyargs[-split:])]

will return (kw, i) tuple for kwonlyargs in reversed order. But if I will pass default value for first and last element, and didn't provide default value for second, third, ... (n-1)th element, then this kwonlyargs_optional will assign optional parameter for (n-1)th element and latest elements, not for first and last. And it will cause an error for typing checking, if you wouldn't provide explicitly kward arg for first kwarg argument.

Possible solution:

kwonlyargs_optional = [(kw, i) for i, kw in enumerate(spec.kwonlydefaults.keys())]

Issue 2:
Checklist

I have verified that the issue exists against the master branch of Celery.

This has already been asked to the discussion group first.

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

#5969

Possible Duplicates

My post

https://stackoverflow.com/questions/66462079/celery-task-does-not-get-send-to-broker

Environment & Settings

Celery version: 5.0.5

celery report Output:

software -> celery:5.0.5 (singularity) kombu:5.0.2 py:3.7.8
billiard:3.6.3.0 py-amqp:5.0.5
platform -> system:Linux arch:64bit
kernel version:4.15.0-20-generic imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:pyamqp results:db+postgresql://docker:**@pg_db:5432/

broker: 'pyamqp://guest@my-rabbit//'
celery_accept_content: ['json']
environ: {
'CELERY_BROKER': 'pyamqp://guest@my-rabbit//',
'CELERY_BROKER_URL': 'amqp://guest:********@my-rabbit:5672//',
'GPG_KEY': '********',
'HOME': '/home/snake',
'LANG': 'C.UTF-8',
'LS_COLORS': 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:',
'PATH': '/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin',
'PWD': '/home/snake',
'PYTHONDONTWRITEBYTECODE': '1',
'PYTHONUNBUFFERED': '1',
'PYTHON_VERSION': '3.7.8',
'SHLVL': '1',
'TERM': 'xterm',
'_': '/usr/local/bin/celery'}
result_backend: 'db+postgresql://docker:********@pg_db:5432/'
result_persistent: False
result_serializer: 'json'
task_serializer: 'json'
worker_concurrency: 16
deprecated_settings: None

Steps to Reproduce

Required Dependencies

Minimal Python Version: 3.7

Minimal Celery Version: 4.7.7

Minimal Kombu Version: 5.0.2

Minimal Broker Version: RabbitMQ

Minimal Result Backend Version: N/A

Minimal OS and/or Kernel Version: 4.15.0-20-generic #21-Ubuntu SMP

Minimal Broker Client Version: N/A or Unknown

Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

alembic==1.5.5
amqp==5.0.5
Beaker==1.11.0
billiard==3.6.3.0
celery==5.0.5
certifi==2020.12.5
cffi==1.14.5
chardet==4.0.0
click==7.1.2
click-didyoumean==0.0.3
click-plugins==1.1.1
click-repl==0.1.6
cryptography==3.4.6
defusedxml==0.7.0
et-xmlfile==1.0.1
Flask==1.1.1
Flask-Migrate==2.5.3
Flask-pyoidc==3.4.0
Flask-SQLAlchemy==2.4.1
Flask-Testing==0.8.1
Flask-WTF==0.14.3
future==0.18.2
gunicorn==20.0.4
idna==2.10
importlib-metadata==3.7.0
importlib-resources==5.1.2
itsdangerous==1.1.0
jdcal==1.4.1
Jinja2==2.11.3
kombu==5.0.2
Mako==1.1.4
MarkupSafe==1.1.1
oic==1.1.2
openpyxl==3.0.3
prompt-toolkit==3.0.16
psycopg2==2.8.6
pycparser==2.20
pycryptodomex==3.10.1
pyjwkest==1.4.2
pyodbc==4.0.30
python-dateutil==2.8.1
python-editor==1.0.4
python-Levenshtein==0.12.2
pytz==2021.1
requests==2.25.1
six==1.15.0
SQLAlchemy==1.3.23
typing-extensions==3.7.4.3
urllib3==1.26.3
vine==5.0.0
wcwidth==0.2.5
Werkzeug==1.0.1
WTForms==2.3.3
zipp==3.4.1

Other Dependencies

N/A

Broker details

Listeners

Interface: [::], port: 15672, protocol: http, purpose: HTTP API
Interface: [::], port: 15692, protocol: http/prometheus, purpose: Prometheus exporter API over HTTP
Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0

Expected Behavior

.delay and .apply_async tasks are send to rabbitmq broker.

Actual Behavior

When I try to send my task to broker (RabbitMQ) it hangs.

# python shell
promise = foo.s(first_arg="2").apply_async()
# blocking indefinitely. I expected a promise object.

If I run the task synchronously it works as expected.

# python shell
promise = foo.s(first_arg="2").apply()
>>> hello argument 2

If I interrupt .apply_async() with ctrl+c I get a traceback with some clues:

Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/kombu/utils/functional.py", line 32, in __call__
return self.__value__
AttributeError: 'ChannelPromise' object has no attribute '__value__'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/amqp/transport.py", line 173, in _connect
host, port, family, socket.SOCK_STREAM, SOL_TCP)
File "/usr/local/lib/python3.7/socket.py", line 752, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -9] Address family for hostname not supported

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/kombu/utils/functional.py", line 325, in retry_over_time
return fun(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/kombu/connection.py", line 866, in _connection_factory
self._connection = self._establish_connection()
File "/usr/local/lib/python3.7/site-packages/kombu/connection.py", line 801, in _establish_connection
conn = self.transport.establish_connection()
File "/usr/local/lib/python3.7/site-packages/kombu/transport/pyamqp.py", line 128, in establish_connection
conn.connect()
File "/usr/local/lib/python3.7/site-packages/amqp/connection.py", line 323, in connect
self.transport.connect()
File "/usr/local/lib/python3.7/site-packages/amqp/transport.py", line 113, in connect
self._connect(self.host, self.port, self.connect_timeout)
File "/usr/local/lib/python3.7/site-packages/amqp/transport.py", line 184, in _connect
"failed to resolve broker hostname"))
File "/usr/local/lib/python3.7/site-packages/amqp/transport.py", line 197, in _connect
self.sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.7/site-packages/celery/canvas.py", line 225, in apply_async
return _apply(args, kwargs, **options)
File "/usr/local/lib/python3.7/site-packages/celery/app/task.py", line 565, in apply_async
**options
File "/usr/local/lib/python3.7/site-packages/celery/app/base.py", line 749, in send_task
amqp.send_task_message(P, name, message, **options)
File "/usr/local/lib/python3.7/site-packages/celery/app/amqp.py", line 532, in send_task_message
**properties
File "/usr/local/lib/python3.7/site-packages/kombu/messaging.py", line 178, in publish
exchange_name, declare,
File "/usr/local/lib/python3.7/site-packages/kombu/connection.py", line 525, in _ensured
return fun(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/kombu/messaging.py", line 184, in _publish
channel = self.channel
File "/usr/local/lib/python3.7/site-packages/kombu/messaging.py", line 206, in _get_channel
channel = self._channel = channel()
File "/usr/local/lib/python3.7/site-packages/kombu/utils/functional.py", line 34, in __call__
value = self.__value__ = self.__contract__()
File "/usr/local/lib/python3.7/site-packages/kombu/messaging.py", line 221, in <lambda>
channel = ChannelPromise(lambda: connection.default_channel)
File "/usr/local/lib/python3.7/site-packages/kombu/connection.py", line 884, in default_channel
self._ensure_connection(**conn_opts)
File "/usr/local/lib/python3.7/site-packages/kombu/connection.py", line 439, in _ensure_connection
callback, timeout=timeout
File "/usr/local/lib/python3.7/site-packages/kombu/utils/functional.py", line 339, in retry_over_time
sleep(1.0)

The broker connection string looks like this in the system:

~$ env | grep BROKER
CELERY_BROKER=pyamqp://guest@172.23.0.3//

The broker connection string in python:

# python shell
from src.celery import app
app.pool.connection
>>> Connection: amqp://guest:**@localhost:5672//

Before you suggest that RabbitMQ is not running, or the connection string is bad; my celery worker (consumer) process is able to connect with the same connection string.

-------------- celery@f9ab48fc6b63 v5.0.5 (singularity)
--- ***** -----
-- ******* ---- Linux-4.15.0-20-generic-x86_64-with-debian-9.12 2021-03-05 07:56:29
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app: celery_statst_api:0x7f15b6de0450
- ** ---------- .> transport: amqp://guest:**@my-rabbit:5672//
- ** ---------- .> results: postgresql://docker:**@pg_db:5432/
- *** --- * --- .> concurrency: 16 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery

[tasks]
. foo_task
. (long list of tasks)

[2021-03-05 07:56:30,564: INFO/MainProcess] Connected to amqp://guest:**@my-rabbit:5672//
[2021-03-05 07:56:30,581: INFO/MainProcess] mingle: searching for neighbors
[2021-03-05 07:56:31,622: INFO/MainProcess] mingle: all alone
[2021-03-05 07:56:31,647: INFO/MainProcess] celery@f9ab48fc6b63 ready.

This is how I connect app/producer to the broker.

The file celeryconfig.py contains setup for broker url backend, concurrency, etc.

# celery_tasks.py
# imports...
app = Celery('celery_statst_api')
app.config_from_object(celeryconfig) # import config file

@app.task(name="foo")
def foo(first_arg: str) -> str:
print(f"thanks for {first_arg}")
return "OK"