Issue 1:
Checklist

I have verified that the issue exists against the master branch of Celery.

This has already been asked to the discussion group first.

-- https://groups.google.com/g/celery-users/c/uqCUtMUu8XY

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

None

Possible Duplicates

None

Environment & Settings

Celery version: 5.0.5 (singularity)

celery report Output:

software -> celery:5.0.5 (singularity) kombu:5.0.2 py:3.7.6
billiard:3.6.3.0 redis:3.5.3
platform -> system:Darwin arch:64bit
kernel version:17.7.0 imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:redis results:redis://localhost:6379/0

broker_url: 'redis://localhost:6379/0'
result_backend: 'redis://lo

Steps to Reproduce

Required Dependencies

Minimal Python Version: N/A or Unknown

Minimal Celery Version: 5.0.0

Minimal Kombu Version: N/A or Unknown

Minimal Broker Version: N/A or Unknown

Minimal Result Backend Version: N/A or Unknown

Minimal OS and/or Kernel Version: N/A or Unknown

Minimal Broker Client Version: N/A or Unknown

Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

amqp==5.0.2
billiard==3.6.3.0
celery @ git+https://github.com/celery/celery.git@491054f2724141cbff20731753379459af033bfd
click==7.1.2
click-didyoumean==0.0.3
click-plugins==1.1.1
click-repl==0.1.6
importlib-metadata==3.3.0
kombu==5.0.2
prompt-toolkit==3.0.8
pytz==2020.4
redis==3.5.3
six==1.15.0
typing-extensions==3.7.4.3
vine==5.0.0
wcwidth==0.2.5
zipp==3.4.0

Other Dependencies

N/A

Minimally Reproducible Test Case

app.py

from celery import Celery

app = Celery(broker='redis://localhost:6379/0', backend='redis://localhost:6379/0')

@app.task(bind=True)
def worker_name(self):
return self.request.hostname

main.py

from app import worker_name

res = worker_name.delay()
print(res.get())

Expected Behavior

As described in the documentation task.request.hostname should be the node name of the worker instance executing the task. See https://docs.celeryproject.org/en/latest/userguide/tasks.html#task-request-info

Actual Behavior

task.request.hostname no longer contains the node name of the worker. Only the hostname of the worker. For example if the worker is started with -n whoopsie@%h only the %h bit is included in task.request.hostname. In fact if the worker is started -n whoopsie@computer it will still only report the hostname of the worker machine and not what has been put into the node name.

This issue appeared between v4.4.7 and v5.0.0 and is present in current master. It is really useful for debugging purposes to know what exact worker instance a task is executed in if there are multiple workers on the same host.

It appears that the worker is at least aware of its worker name based on the worker logs:

celery -A app.app worker -n whoopsie@computer --loglevel=DEBUG

[2020-12-22 10:40:06,971: DEBUG/MainProcess] | Worker: Preparing bootsteps.
[2020-12-22 10:40:06,973: DEBUG/MainProcess] | Worker: Building graph...
[2020-12-22 10:40:06,974: DEBUG/MainProcess] | Worker: New boot order: {Timer, Hub, Pool, Autoscaler, StateDB, Beat, Consumer}
[2020-12-22 10:40:06,986: DEBUG/MainProcess] | Consumer: Preparing bootsteps.
[2020-12-22 10:40:06,987: DEBUG/MainProcess] | Consumer: Building graph...
[2020-12-22 10:40:07,014: DEBUG/MainProcess] | Consumer: New boot order: {Connection, Events, Heart, Mingle, Tasks, Control, Agent, Gossip, event loop}

-------------- whoopsie@computer v5.0.5 (singularity)
--- ***** -----
-- ******* ---- Darwin-17.7.0-x86_64-i386-64bit 2020-12-22 10:40:07
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app: __main__:0x10f0db990
- ** ---------- .> transport: redis://localhost:6379/0
- ** ---------- .> results: redis://localhost:6379/0
- *** --- * --- .> concurrency: 4 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery

[tasks]
. app.worker_name
. celery.accumulate
. celery.backend_cleanup
. celery.chain
. celery.chord
. celery.chord_unlock
. celery.chunks
. celery.group
. celery.map
. celery.starmap

[2020-12-22 10:40:07,041: DEBUG/MainProcess] | Worker: Starting Hub
[2020-12-22 10:40:07,041: DEBUG/MainProcess] ^-- substep ok
[2020-12-22 10:40:07,042: DEBUG/MainProcess] | Worker: Starting Pool
[2020-12-22 10:40:07,237: DEBUG/MainProcess] ^-- substep ok
[2020-12-22 10:40:07,238: DEBUG/MainProcess] | Worker: Starting Consumer
[2020-12-22 10:40:07,239: DEBUG/MainProcess] | Consumer: Starting Connection
[2020-12-22 10:40:07,286: INFO/MainProcess] Connected to redis://localhost:6379/0
[2020-12-22 10:40:07,287: DEBUG/MainProcess] ^-- substep ok
[2020-12-22 10:40:07,287: DEBUG/MainProcess] | Consumer: Starting Events
[2020-12-22 10:40:07,310: DEBUG/MainProcess] ^-- substep ok
[2020-12-22 10:40:07,311: DEBUG/MainProcess] | Consumer: Starting Heart
[2020-12-22 10:40:07,317: DEBUG/MainProcess] ^-- substep ok
[2020-12-22 10:40:07,317: DEBUG/MainProcess] | Consumer: Starting Mingle
[2020-12-22 10:40:07,317: INFO/MainProcess] mingle: searching for neighbors
[2020-12-22 10:40:08,397: INFO/MainProcess] mingle: all alone
[2020-12-22 10:40:08,397: DEBUG/MainProcess] ^-- substep ok
[2020-12-22 10:40:08,397: DEBUG/MainProcess] | Consumer: Starting Tasks
[2020-12-22 10:40:08,405: DEBUG/MainProcess] ^-- substep ok
[2020-12-22 10:40:08,405: DEBUG/MainProcess] | Consumer: Starting Control
[2020-12-22 10:40:08,422: DEBUG/MainProcess] ^-- substep ok
[2020-12-22 10:40:08,423: DEBUG/MainProcess] | Consumer: Starting Gossip
[2020-12-22 10:40:08,446: DEBUG/MainProcess] ^-- substep ok
[2020-12-22 10:40:08,446: DEBUG/MainProcess] | Consumer: Starting event loop
[2020-12-22 10:40:08,446: DEBUG/MainProcess] | Worker: Hub.register Pool...
[2020-12-22 10:40:08,447: INFO/MainProcess] whoopsie@computer ready.
[2020-12-22 10:40:08,448: DEBUG/MainProcess] basic.qos: prefetch_count->16
[2020-12-22 10:40:16,492: INFO/MainProcess] Received task: app.worker_name[d5f96484-0a00-4726-ac54-222765aa0901]
[2020-12-22 10:40:16,493: DEBUG/MainProcess] TaskPool: Apply <function _trace_task_ret at 0x10f05b3b0> (args:('app.worker_name', 'd5f96484-0a00-4726-ac54-222765aa0901', {'lang': 'py', 'task': 'app.worker_name', 'id': 'd5f96484-0a00-4726-ac54-222765aa0901', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'group_index': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'd5f96484-0a00-4726-ac54-222765aa0901', 'parent_id': None, 'argsrepr': '()', 'kwargsrepr': '{}', 'origin': 'gen3304@Fredriks-MacBook-Pro.local', 'reply_to': 'a0afc794-880a-3e05-970f-5013e0b0ed61', 'correlation_id': 'd5f96484-0a00-4726-ac54-222765aa0901', 'hostname': 'whoopsie@computer', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': None}, 'args': [], 'kwargs': {}}, b'[[], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
[2020-12-22 10:40:16,495: DEBUG/MainProcess] Task accepted: app.worker_name[d5f96484-0a00-4726-ac54-222765aa0901] pid:3300
[2020-12-22 10:40:16,513: INFO/ForkPoolWorker-2] Task app.worker_name[d5f96484-0a00-4726-ac54-222765aa0901] succeeded in 0.016767310000000535s: 'Fredriks-MacBook-Pro.local'

Issue 2:
If for some reason the private key used as broker_use_ssl.keyfile is not an RSA private key, Celery will crash without telling the root cause, leading to difficult debugging. This affects current celery master branch.

This is due to celery.security.key.PrivateKey __init__ method not checking the returned key type from load_pem_private_key (from cryptography.hazmat). If the returned key is, for example, EllipticCurvePrivateKey, the sign method of PrivateKey later will crash with TypeError: sign() takes 3 positional arguments but 4 were given as it expects that self._key is instance of RSAPrivateKey. Furthermore, this (opaque) exception is followed by many more exceptions of type "TypeError: catching classes that do not inherit from BaseException is not allowed". See reproduction for a complete crash log.

cryptography library load_pem_private_key documentation explicitly states that "Many serialization formats support multiple different types of asymmetric keys and will return an instance of the appropriate type. You should check that the returned key matches the type your application expects when using these methods."

I propose that the type of the returned key should be checked and appropriate errors raised if the key type is not supported. This would inform the user that the problem they are facing is not a bug in celery or their implementation, but rather that their certificates are not supported by celery.

Checklist

I have verified that the issue exists against the master branch of Celery.

This has already been asked to the discussions forum first.

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Related Issues

None that I could find

Possible Duplicates

None that I could find

Environment & Settings

$ celery --version
5.0.5 (singularity)

cryptography==37.0.4

Steps to Reproduce

app = Celery('test', include=[])
broker_use_ssl = {
'keyfile': 'broker_client.key', # private key is ECDSA signed
'certfile': 'broker_client.crt',
'ca_certs': 'root.pem',
'cert_reqs': ssl.CERT_REQUIRED
}

app.conf.update(
broker_use_ssl=broker_use_ssl,
security_key='private.key',
security_certificate='public.crt',
security_cert_store='security_store/*.pem
)

app.setup_security()
app.start()

This should result in traceback

[2022-07-22 14:16:20,309: CRITICAL/MainProcess] Unrecoverable error: EncodeError(SecurityError("Unable to serialize: TypeError('catching classes that do not inherit from BaseException is not allowed')"))

Traceback (most recent call last):
File "/venv/lib/python3.8/site-packages/celery/security/utils.py", line 24, in reraise_errors
yield
File "/venv/lib/python3.8/site-packages/celery/security/key.py", line 32, in sign
return self._key.sign(ensure_bytes(data), padd, digest)
TypeError: sign() takes 3 positional arguments but 4 were given
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/venv/lib/python3.8/site-packages/celery/security/utils.py", line 24, in reraise_errors
yield
File "/venv/lib/python3.8/site-packages/celery/security/serialization.py", line 39, in serialize
signature=self._key.sign(body, self._digest),
File "/venv/lib/python3.8/site-packages/celery/security/key.py", line 32, in sign
return self._key.sign(ensure_bytes(data), padd, digest)
File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
self.gen.throw(type, value, traceback)
File "/venv/lib/python3.8/site-packages/celery/security/utils.py", line 25, in reraise_errors
except errors as exc:
TypeError: catching classes that do not inherit from BaseException is not allowed
... Above type error repeated many times ...