Issue 1:
The type of traceback is always string, and children is list.

And An error occurred when decode it.

It tried to set serializer to json or pickle, and Their results are the same.

Thanks.

$ celery --version

5.0.2 (singularity)

Related Issues and Possible Duplicates

#6364

File "msgpack_unpacker.pyx", line 178, in msgpack._cmsgpack.unpackb

File "msgpack_unpacker.pyx", line 126, in msgpack._cmsgpack.get_data_from_buffer

kombu.exceptions.DecodeError: a bytes-like object is required, not 'str'

celery/celery/backends/mongodb.py

Line 205
in
60ba379

'traceback': self.decode(obj['traceback']),

return self.meta_from_decoded({
'task_id': obj['_id'],
'status': obj['status'],
'result': self.decode(obj['result']),
'date_done': obj['date_done'],
'traceback': self.decode(obj['traceback']),
'children': self.decode(obj['children']),
})

Issue 2:
Checklist

I have verified that the issue exists against the master branch of Celery.

This has already been asked to the discussions forum first.

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

#6661

Possible Duplicates

#6661

Environment & Settings

Celery version: 5.2.5

celery report Output:

software -> celery:5.2.5 (dawn-chorus) kombu:5.2.4 py:3.8.13

billiard:3.6.4.0 py-amqp:5.1.0

platform -> system:Linux arch:64bit

kernel version:5.4.0-1069-aws imp:CPython

loader -> celery.loaders.default.Loader

settings -> transport:amqp results:disabled

deprecated_settings: None

Steps to Reproduce

Required Dependencies

Minimal Python Version: 3.8

Minimal Celery Version: 5.2.3

Minimal Kombu Version: 5.2.4

Minimal Broker Version: Redis

Minimal Result Backend Version: N/A

Minimal OS and/or Kernel Version: Debian GNU/Linux 10 (buster)

Minimal Broker Client Version: N/A or Unknown

Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

alembic==1.7.7

amqp==5.1.0

atomicwrites==1.3.0

attrs==19.3.0

baseplate==2.1.1

billiard==3.6.4.0

boto3==1.21.35

botocore==1.24.38

celery==5.2.5

certifi==2020.12.5

cffi==1.14.5

chardet==4.0.0

click==8.1.2

click-didyoumean==0.3.0

click-plugins==1.1.1

click-repl==0.2.0

coverage==5.0

cryptography==3.3.2

gevent==21.1.2

greenlet==1.0.0

hiredis==2.0.0

idna==2.10

importlib-metadata==1.7.0

importlib-resources==5.6.0

jmespath==1.0.0

kombu==5.2.4

Mako==1.2.0

MarkupSafe==2.1.1

more-itertools==7.2.0

mypy==0.910

mypy-extensions==0.4.3

packaging==19.1

pluggy==0.12.0

posix-ipc==1.0.5

Editable install with no version control (product-catalog==0.0.0)

-e /src

prompt-toolkit==3.0.29

psycogreen==1.0.1

psycopg2==2.8.5

py==1.10.0

pycparser==2.20

PyJWT==2.0.1

pyparsing==2.4.6

pytest==5.1.2

pytest-cov==2.6.1

pytest-random-order==1.0.4

python-dateutil==2.8.2

python-json-logger==2.0.1

pytz==2022.1

reddit-edgecontext==1.0.0

redis==3.5.3

requests==2.25.1

s3transfer==0.5.2

sentry-sdk==0.20.3

simpleflake==0.1.5

six==1.15.0

SQLAlchemy==1.3.19

thrift-unofficial==0.14.1

toml==0.10.2

typed-ast==1.4.1

typing-extensions==3.7.4.2

urllib3==1.26.5

vine==5.0.0

wcwidth==0.1.7

zipp==3.1.0

zope.event==4.5.0

zope.interface==5.2.0

Other Dependencies

N/A

Expected Behavior

.delay sending task method from app's docker container to worker's container produces connection error,

however both containers are deployed correctly and working.

Actual Behavior

I have following code reimplementing original Celery and Task classes in separate app.py:

def short_task_name(task_name: str) -> str:
return task_name.split(".")[-1]

def emit_celery_metric(context: baseplate.RequestContext, task_name: str, **tags: Any) -> None:
tags = {"task_name": short_task_name(task_name), **tags}
context.metrics.counter("celery_task", tags=tags).increment()
context.metrics.flush()

class BaseplateTask(celery.Task):
# this lets us get away with injecting the context task arg
typing = False

def __call__(self, *args, **kwargs) -> None:
baseplate = self._app._baseplate
context = baseplate.make_context_object()

with baseplate.make_server_span(context, f"celery.{short_task_name(self.name)}"):
try:
ret = super().__call__(context, *args, **kwargs)
except Exception:
emit_celery_metric(context, self.name, success=False)
raise
emit_celery_metric(context, self.name, success=True)

return ret

class BaseplateCelery(celery.Celery):
def __init__(self, *args, **kwargs) -> None:
self._baseplate = None
super().__init__(*args, **kwargs)

def set_baseplatel(self, baseplate, config) -> None:
self._baseplate = baseplate
self.conf.update(broker_url=config['broker_url'])

def run_workers(self, config: baseplate.lib.config.RawConfig) -> None:
"""Start the worker, sets up context for future use in workers."""
logger.info("Starting celery workers")

self.start(config["celery_worker_command"].split(" "))

celery = BaseplateCelery("product_catalog") # I can't pass broker url directly here, because I have to take it from config and set later
celery.Task = BaseplateTask

Here enough just a simple task to reproduce the error case in tasks.py:

from . import app

@app.celery.task
def test(context):
print(type(context))

Just a func in call.py to get caledl from the shell:

from . import tasks

def foo():
tasks.test.get(context) # context here will be passed in reimplemented Task class

Then main function in __main__.py is for initialization:

import baseplate

from . import app
from . import command

def main():
baseplate = baseplate.Baseplate()
app.celery.set_baseplate(baseplate, config={'broker_url': 'redis://redis:6379'}) # broker url is correct, works
app.celery.run_workers(command.config)

After both containers get deployed, I can reproduces the behavior via shell:

from call import foo
foo()

It gives me a stack trace:

Traceback (most recent call last):
File "/usr/local/lib/venv/lib/python3.8/site-packages/kombu/utils/functional.py", line 30, in __call__
return self.__value__
AttributeError: 'ChannelPromise' object has no attribute '__value__'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/venv/lib/python3.8/site-packages/amqp/transport.py", line 188, in _connect
entries = socket.getaddrinfo(
File "/usr/local/lib/python3.8/socket.py", line 918, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -9] Address family for hostname not supported

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/venv/lib/python3.8/site-packages/kombu/connection.py", line 446, in _reraise_as_library_errors
yield
File "/usr/local/lib/venv/lib/python3.8/site-packages/kombu/connection.py", line 433, in _ensure_connection
return retry_over_time(
File "/usr/local/lib/venv/lib/python3.8/site-packages/kombu/utils/functional.py", line 312, in retry_over_time
return fun(*args, **kwargs)
File "/usr/local/lib/venv/lib/python3.8/site-packages/kombu/connection.py", line 877, in _connection_factory
self._connection = self._establish_connection()
File "/usr/local/lib/venv/lib/python3.8/site-packages/kombu/connection.py", line 812, in _establish_connection
conn = self.transport.establish_connection()
File "/usr/local/lib/venv/lib/python3.8/site-packages/kombu/transport/pyamqp.py", line 201, in establish_connection
conn.connect()
File "/usr/local/lib/venv/lib/python3.8/site-packages/amqp/connection.py", line 323, in connect
self.transport.connect()
File "/usr/local/lib/venv/lib/python3.8/site-packages/amqp/transport.py", line 129, in connect
self._connect(self.host, self.port, self.connect_timeout)
File "/usr/local/lib/venv/lib/python3.8/site-packages/amqp/transport.py", line 197, in _connect
raise (e
File "/usr/local/lib/venv/lib/python3.8/site-packages/amqp/transport.py", line 213, in _connect
self.sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/src/product_catalog/__main__", line 81, in main
tasks.foo.delay()
File "/usr/local/lib/venv/lib/python3.8/site-packages/celery/app/task.py", line 425, in delay
return self.apply_async(args, kwargs)
File "/usr/local/lib/venv/lib/python3.8/site-packages/celery/app/task.py", line 575, in apply_async
return app.send_task(
File "/usr/local/lib/venv/lib/python3.8/site-packages/celery/app/base.py", line 788, in send_task
amqp.send_task_message(P, name, message, **options)
File "/usr/local/lib/venv/lib/python3.8/site-packages/celery/app/amqp.py", line 510, in send_task_message
ret = producer.publish(
File "/usr/local/lib/venv/lib/python3.8/site-packages/kombu/messaging.py", line 177, in publish
return _publish(
File "/usr/local/lib/venv/lib/python3.8/site-packages/kombu/connection.py", line 523, in _ensured
return fun(*args, **kwargs)
File "/usr/local/lib/venv/lib/python3.8/site-packages/kombu/messaging.py", line 186, in _publish
channel = self.channel
File "/usr/local/lib/venv/lib/python3.8/site-packages/kombu/messaging.py", line 209, in _get_channel
channel = self._channel = channel()
File "/usr/local/lib/venv/lib/python3.8/site-packages/kombu/utils/functional.py", line 32, in __call__
value = self.__value__ = self.__contract__()
File "/usr/local/lib/venv/lib/python3.8/site-packages/kombu/messaging.py", line 225, in <lambda>
channel = ChannelPromise(lambda: connection.default_channel)
File "/usr/local/lib/venv/lib/python3.8/site-packages/kombu/connection.py", line 895, in default_channel
self._ensure_connection(**conn_opts)
File "/usr/local/lib/venv/lib/python3.8/site-packages/kombu/connection.py", line 433, in _ensure_connection
return retry_over_time(
File "/usr/local/lib/python3.8/contextlib.py", line 131, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/local/lib/venv/lib/python3.8/site-packages/kombu/connection.py", line 450, in _reraise_as_library_errors
raise ConnectionError(str(exc)) from exc
kombu.exceptions.OperationalError: [Errno 111] Connection refused

That is very strange. Because celery is running in container:

-------------- celery@rcelery-workers-7d4d5bd5c8-4slh5 v5.2.5 (dawn-chorus)
--- ***** -----
-- ******* ---- Linux-5.4.0-1069-aws-x86_64-with-glibc2.2.5 2022-04-12 11:56:16
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app: __main__:0x7f0df9a15a30
- ** ---------- .> transport: redis://redis:6379/0
- ** ---------- .> results: disabled://
- *** --- * --- .> concurrency: 1 (gevent)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery

[tasks]
. product_catalog.celery.tasks.test

[2022-04-12 11:56:16,208: INFO/MainProcess] Connected to redis://redis:6379/0
[2022-04-12 11:56:16,211: INFO/MainProcess] mingle: searching for neighbors
[2022-04-12 11:56:17,226: INFO/MainProcess] mingle: sync with 1 nodes
[2022-04-12 11:56:17,226: INFO/MainProcess] mingle: sync complete
[2022-04-12 11:56:17,234: INFO/MainProcess] pidbox: Connected to redis://redis:6379/0.
[2022-04-12 11:56:17,241: INFO/MainProcess] celery@celery-workers-7d4d5bd5c8-4slh5 ready.
[2022-04-12 11:56:17,632: WARNING/MainProcess] Substantial drift from celery@ip-192-168-0-16.ec2.internal may mean clocks are out of sync. Current drift is 21600 seconds. [orig: 2022-04-12 11:56:17.632010 recv: 2022-04-12 17:56:17.339628]
[2022-04-12 11:56:52,239: INFO/MainProcess] missed heartbeat from celery@celery-workers-6f67f5499b-vp22v

As I already commented in code, I can't pass broker url to init constructor due to some restrictions from top.

But if it was passed to constructor without setting afterwards in self.conf, then everything works well and task will be sent to worker and processed properly! But I have to initialize the app first, and then set broker url.

I suspect that problem is in decorator, because when I initialize celery app without broker set, this task gets wrapped with default broker address to use. That's why kombu uses the default broker address(not sure).

Although I tried to patch BaseplateCelery with additional method to call later:

def call_super(self):
super().__init__(main='product_catalog', broker_url='redis://redis:6379/0')

i see that class state now is if it was initialized in working case, but it still produces the same error!