Issue 1:
Checklist

[ X] I have verified that the issue exists against the master branch of Celery.

[ X] This has already been asked to the discussion group first.

[ X] I have read the relevant section in the

contribution guide

on reporting bugs.

[ X] I have checked the issues list

for similar or identical bug reports.

[ X] I have checked the pull requests list

for existing proposed fixes.

[ X] I have checked the commit log

to find out if the bug was already fixed in the master branch.

[ X] I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

[ X] I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

[ X] I have verified that the issue exists against the master branch of Celery.

[ X] I have included the contents of pip freeze in the issue.

[ X] I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

None

Possible Duplicates

None

Environment & Settings

Celery version:

software -> celery:5.0.2 (singularity) kombu:5.0.2 py:3.7.4
billiard:3.6.3.0 py-amqp:5.0.2
platform -> system:Linux arch:64bit, ELF
kernel version:5.9.16_1 imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:amqp results:rpc:///

broker_url: 'amqp://guest:********@localhost:5672//'

result_backend: 'rpc:///'

deprecated_settings: None

Output:

Steps to Reproduce

Required Dependencies

Minimal Python Version: N/A or Unknown

Minimal Celery Version: N/A or Unknown

Minimal Kombu Version: N/A or Unknown

Minimal Broker Version: N/A or Unknown

Minimal Result Backend Version: N/A or Unknown

Minimal OS and/or Kernel Version: N/A or Unknown

Minimal Broker Client Version: N/A or Unknown

Minimal Result Backend Client Version: N/A or Unknown

Python Packages

asn1crypto==0.24.0
brotlipy==0.7.0
certifi==2020.6.20
cffi==1.14.0
chardet==3.0.4
conda==4.8.3
conda-package-handling==1.7.0
configparser==3.7.4
cryptography==2.9.2
idna @ file:///tmp/build/80754af9/idna_1593446292537/work
OWSLib==0.19.2
pycosat==0.6.3
pycparser @ file:///tmp/build/80754af9/pycparser_1594388511720/work
pyOpenSSL @ file:///tmp/build/80754af9/pyopenssl_1594392929924/work
pyproj==2.5.0
PySocks @ file:///tmp/build/80754af9/pysocks_1594394576006/work
python-dateutil==2.8.1
pytz==2019.3
PyYAML==5.3
requests @ file:///tmp/build/80754af9/requests_1592841827918/work
ruamel-yaml==0.15.87
six==1.15.0
tqdm @ file:///tmp/build/80754af9/tqdm_1593446365756/work
urllib3==1.25.9

Output:

Other Dependencies

# packages in environment at /opt/miniconda3/envs/[redacted]:
#
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 1_gnu conda-forge
aggdraw 1.3.12 py37hdf465ad_1 conda-forge
amqp 5.0.2 pyhd8ed1ab_0 conda-forge
appdirs 1.4.4 pyh9f0ad1d_0 conda-forge
argon2-cffi 20.1.0 py37h4abf009_2 conda-forge
asgiref 3.3.1 pyhd8ed1ab_0 conda-forge
astroid 2.4.2 py37hc8dfbb8_1 conda-forge
async_generator 1.10 py_0 conda-forge
attrs 20.3.0 pyhd3deb0d_0 conda-forge
autopep8 1.5.4 pyh9f0ad1d_0 conda-forge
azure-core 1.9.0 pypi_0 pypi
azure-storage-blob 12.6.0 pypi_0 pypi
backcall 0.2.0 pyh9f0ad1d_0 conda-forge
backports 1.0 py_2 conda-forge
backports.functools_lru_cache 1.6.1 py_0 conda-forge
billiard 3.6.3.0 py37h8f50634_2 conda-forge
black 20.8b1 py_1 conda-forge
bleach 3.2.1 pyh9f0ad1d_0 conda-forge
boost-cpp 1.72.0 h8e57a91_0 conda-forge
brotlipy 0.7.0 py37hb5d75c8_1001 conda-forge
bzip2 1.0.8 h7f98852_4 conda-forge
c-ares 1.17.1 h36c2ea0_0 conda-forge
ca-certificates 2020.12.5 ha878542_0 conda-forge
cairocffi 1.2.0 pypi_0 pypi
cairosvg 2.5.0 pypi_0 pypi
celery 5.0.2 pyhd8ed1ab_0 conda-forge
certifi 2020.12.5 py37h89c1867_1 conda-forge
cffi 1.14.4 py37h11fe52a_0 conda-forge
chardet 4.0.0 py37h89c1867_0 conda-forge
click 7.1.2 pyh9f0ad1d_0 conda-forge
click-didyoumean 0.0.3 pyh8c360ce_0 conda-forge
click-repl 0.1.6 py_0 conda-forge
colorzero 1.1 pypi_0 pypi
cryptography 3.3.1 py37h7f0c10b_0 conda-forge
cssselect2 0.4.1 pypi_0 pypi
cycler 0.10.0 pypi_0 pypi
dataclasses 0.7 pyhb2cacf7_7 conda-forge
dbus 1.13.6 hfdff14a_1 conda-forge
decorator 4.4.2 py_0 conda-forge
defusedxml 0.6.0 py_0 conda-forge
django 3.1.5 pyhd8ed1ab_0 conda-forge
django-crispy-forms 1.10.0 pypi_0 pypi
drawsvg 1.7.0 pypi_0 pypi
entrypoints 0.3 pyhd8ed1ab_1003 conda-forge
expat 2.2.9 he1b5a44_2 conda-forge
flake8 3.8.4 py_0 conda-forge
fontconfig 2.13.1 h86ecdb6_1001 conda-forge
freeimageplus 3.18.0 hf484d3e_2 dlr-sc
freetype 2.10.4 h7ca028e_0 conda-forge
future 0.18.2 py37h89c1867_2 conda-forge
gettext 0.19.8.1 hf34092f_1004 conda-forge
glib 2.66.3 h58526e2_0 conda-forge
gmp 6.2.1 h58526e2_0 conda-forge
gmsh 4.6.0 h7d08f12_0 conda-forge
gpiozero 1.5.1 pypi_0 pypi
gst-plugins-base 1.14.5 h0935bb2_2 conda-forge
gstreamer 1.14.5 h36ae1b5_2 conda-forge
guizero 1.1.1 pypi_0 pypi
h11 0.12.0 pypi_0 pypi
icu 64.2 he1b5a44_1 conda-forge
idna 2.10 pyh9f0ad1d_0 conda-forge
ifcopenshell v0.6.0 py37h4488930_3 conda-forge
imageio 2.9.0 pypi_0 pypi
importlib-metadata 3.3.0 py37h89c1867_2 conda-forge
importlib_metadata 3.3.0 hd8ed1ab_2 conda-forge
ipykernel 5.4.2 py37h888b3d9_0 conda-forge
ipython 7.19.0 py37h888b3d9_1 conda-forge
ipython_genutils 0.2.0 py_1 conda-forge
ipywidgets 7.6.2 pyhd3deb0d_0 conda-forge
isodate 0.6.0 pypi_0 pypi
isort 5.7.0 pyhd8ed1ab_0 conda-forge
jedi 0.17.2 py37h89c1867_1 conda-forge
jinja2 2.11.2 pyh9f0ad1d_0 conda-forge
jpeg 9d h36c2ea0_0 conda-forge
jsonschema 3.2.0 py_2 conda-forge
jupyter 1.0.0 py_2 conda-forge
jupyter_client 6.1.7 py_0 conda-forge
jupyter_console 6.2.0 py_0 conda-forge
jupyter_core 4.7.0 py37h89c1867_0 conda-forge
jupyterlab_pygments 0.1.2 pyh9f0ad1d_0 conda-forge
jupyterlab_widgets 1.0.0 pyhd8ed1ab_1 conda-forge
kiwisolver 1.3.1 pypi_0 pypi
kombu 5.0.2 py37h89c1867_1 conda-forge
krb5 1.17.2 h926e7f8_0 conda-forge
lazy-object-proxy 1.4.3 py37h8f50634_2 conda-forge
lcms2 2.11 hcbb858e_1 conda-forge
libblas 3.9.0 6_openblas conda-forge
libclang 9.0.1 default_hde54327_0 conda-forge
libcurl 7.71.1 hcdd3856_8 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 h516909a_1 conda-forge
libffi 3.2.1 he1b5a44_1007 conda-forge
libgcc-ng 9.3.0 h5dbcf3e_17 conda-forge
libgfortran-ng 9.3.0 he4bcb1c_17 conda-forge
libgfortran5 9.3.0 he4bcb1c_17 conda-forge
libglib 2.66.3 hbe7bbb4_0 conda-forge
libglu 9.0.0 he1b5a44_1001 conda-forge
libgomp 9.3.0 h5dbcf3e_17 conda-forge
libiconv 1.16 h516909a_0 conda-forge
liblapack 3.9.0 6_openblas conda-forge
libllvm9 9.0.1 hf817b99_2 conda-forge
libnghttp2 1.41.0 h8cfc5f6_2 conda-forge
libopenblas 0.3.12 pthreads_h4812303_1 conda-forge
libpng 1.6.37 h21135ba_2 conda-forge
libsodium 1.0.18 h36c2ea0_1 conda-forge
libssh2 1.9.0 hab1572f_5 conda-forge
libstdcxx-ng 9.3.0 h2ae2ef3_17 conda-forge
libtiff 4.2.0 hdc55705_0 conda-forge
libuuid 2.32.1 h7f98852_1000 conda-forge
libwebp-base 1.1.0 h36c2ea0_3 conda-forge
libxcb 1.13 h14c3975_1002 conda-forge
libxkbcommon 0.10.0 he1b5a44_0 conda-forge
libxml2 2.9.10 hee79883_0 conda-forge
libxslt 1.1.33 h31b3aaa_0 conda-forge
lz4-c 1.9.3 h9c3ff4c_0 conda-forge
markupsafe 1.1.1 py37hb5d75c8_2 conda-forge
matplotlib 2.2.2 pypi_0 pypi
mccabe 0.6.1 py_1 conda-forge
mistune 0.8.4 py37h4abf009_1002 conda-forge
mono 5.14.0.177 he1b5a44_1001 conda-forge
msrest 0.6.19 pypi_0 pypi
mu-editor 1.0.3 pypi_0 pypi
mypy_extensions 0.4.3 py37h89c1867_2 conda-forge
nbclient 0.5.1 py_0 conda-forge
nbconvert 6.0.7 py37h89c1867_3 conda-forge
nbformat 5.0.8 py_0 conda-forge
ncurses 6.2 h58526e2_4 conda-forge
nest-asyncio 1.4.3 pyhd8ed1ab_0 conda-forge
notebook 6.1.6 py37h89c1867_0 conda-forge
nspr 4.29 h9c3ff4c_1 conda-forge
nss 3.59 h2c00c37_0 conda-forge
nudatus 0.0.4 pypi_0 pypi
numpy 1.19.4 pypi_0 pypi
oauthlib 3.1.0 pypi_0 pypi
occt 7.4.0 h9121d39_3 conda-forge
olefile 0.46 pyh9f0ad1d_1 conda-forge
opencascade 7.4.0 h6bb024c_5 dlr-sc
openssl 1.1.1i h7f98852_0 conda-forge
owslib 0.21.0 pyhd8ed1ab_0 conda-forge
packaging 20.8 pyhd3deb0d_0 conda-forge
pandoc 2.11.3.2 h7f98852_0 conda-forge
pandocfilters 1.4.2 py_1 conda-forge
parso 0.7.1 pyh9f0ad1d_0 conda-forge
pathspec 0.8.1 pyhd3deb0d_0 conda-forge
pcre 8.44 he1b5a44_0 conda-forge
pexpect 4.8.0 pyh9f0ad1d_2 conda-forge
pgzero 1.2 pypi_0 pypi
pickleshare 0.7.5 py_1003 conda-forge
pigpio 1.78 pypi_0 pypi
pillow 8.1.0 py37he6b4880_0 conda-forge
pip 20.3.3 pyhd8ed1ab_0 conda-forge
pluggy 0.13.1 py37he5f6b98_3 conda-forge
proj 7.2.0 h8b9fe22_0 conda-forge
prometheus_client 0.9.0 pyhd3deb0d_0 conda-forge
prompt-toolkit 3.0.8 pyha770c72_0 conda-forge
prompt_toolkit 3.0.8 hd8ed1ab_0 conda-forge
pthread-stubs 0.4 h36c2ea0_1001 conda-forge
ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge
punycodes 1.0.1 pypi_0 pypi
pycodestyle 2.4.0 pypi_0 pypi
pycparser 2.20 pyh9f0ad1d_2 conda-forge
pydocstyle 5.1.1 py_0 conda-forge
pyflakes 2.0.0 pypi_0 pypi
pygame 1.9.6 pypi_0 pypi
pygments 2.7.3 pyhd8ed1ab_0 conda-forge
pylint 2.6.0 py37hc8dfbb8_1 conda-forge
pyls 0.1.6 pypi_0 pypi
pyopenssl 20.0.1 pyhd8ed1ab_0 conda-forge
pyparsing 2.4.7 pyh9f0ad1d_0 conda-forge
pyproj 3.0.0.post1 py37hf874b97_0 conda-forge
pyqt5 5.14.1 pypi_0 pypi
pyqt5-sip 12.8.1 pypi_0 pypi
pyqtchart 5.14.0 pypi_0 pypi
pyqtwebengine 5.12.1 pypi_0 pypi
pyrsistent 0.17.3 py37h4abf009_1 conda-forge
pyserial 3.4 pypi_0 pypi
pyside2 5.13.2 py37hbee1fde_2 conda-forge
pysocks 1.7.1 py37he5f6b98_2 conda-forge
python 3.7.4 h265db76_1
python-dateutil 2.8.1 py_0 conda-forge
python-jsonrpc-server 0.4.0 pyh9f0ad1d_0 conda-forge
python-language-server 0.36.2 pyhd8ed1ab_0 conda-forge
python_abi 3.7 1_cp37m conda-forge
pythonnet 2.4.0 py37haf93ef1_1 conda-forge
pythonocc-core 7.4.1 py37h2bc3f7f_0 dlr-sc
pytz 2020.5 pyhd8ed1ab_0 conda-forge
pyyaml 5.3.1 py37hb5d75c8_1 conda-forge
pyzmq 20.0.0 py37h5a562af_1 conda-forge
qscintilla 2.11.4 pypi_0 pypi
qt 5.12.5 hd8c4c69_1 conda-forge
qtconsole 4.3.1 pypi_0 pypi
qtpy 1.9.0 py_0 conda-forge
readline 7.0 hf8c457e_1001 conda-forge
regex 2020.11.13 py37h5e8e339_0 conda-forge
reportlab 3.5.59 py37h69800bb_0 conda-forge
requests 2.25.1 pyhd3deb0d_0 conda-forge
requests-oauthlib 1.3.0 pypi_0 pypi
rope 0.18.0 pyh9f0ad1d_0 conda-forge
semver 2.13.0 pypi_0 pypi
send2trash 1.5.0 py_0 conda-forge
setuptools 49.6.0 py37he5f6b98_2 conda-forge
six 1.15.0 pyh9f0ad1d_0 conda-forge
snowballstemmer 2.0.0 py_0 conda-forge
sqlalchemy 1.3.22 py37h5e8e339_1 conda-forge
sqlite 3.33.0 h62c20be_0
sqlparse 0.4.1 pyh9f0ad1d_0 conda-forge
tbb 2020.2 h4bd325d_2 conda-forge
terminado 0.9.1 py37h89c1867_1 conda-forge
testpath 0.4.4 py_0 conda-forge
tinycss2 1.1.0 pypi_0 pypi
tk 8.6.10 h21135ba_1 conda-forge
toml 0.10.2 pyhd8ed1ab_0 conda-forge
tornado 6.1 py37h4abf009_0 conda-forge
traitlets 5.0.5 py_0 conda-forge
typed-ast 1.4.2 py37h5e8e339_0 conda-forge
typing_extensions 3.7.4.3 py_0 conda-forge
ujson 4.0.1 py37h3340039_1 conda-forge
urllib3 1.26.2 pyhd8ed1ab_0 conda-forge
uvicorn 0.13.3 pypi_0 pypi
vine 5.0.0 pyhd8ed1ab_1 conda-forge
wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge
webencodings 0.5.1 py_1 conda-forge
wheel 0.36.2 pyhd3deb0d_0 conda-forge
widgetsnbextension 3.5.1 py37h89c1867_4 conda-forge
wrapt 1.11.2 py37h8f50634_1 conda-forge
xorg-fixesproto 5.0 h14c3975_1002 conda-forge
xorg-kbproto 1.0.7 h14c3975_1002 conda-forge
xorg-libice 1.0.10 h516909a_0 conda-forge
xorg-libsm 1.2.3 h84519dc_1000 conda-forge
xorg-libx11 1.6.12 h516909a_0 conda-forge
xorg-libxau 1.0.9 h14c3975_0 conda-forge
xorg-libxdmcp 1.1.3 h516909a_0 conda-forge
xorg-libxext 1.3.4 h516909a_0 conda-forge
xorg-libxfixes 5.0.3 h516909a_1004 conda-forge
xorg-libxmu 1.1.3 h516909a_0 conda-forge
xorg-libxrender 0.9.10 h516909a_1002 conda-forge
xorg-libxt 1.1.5 h516909a_1003 conda-forge
xorg-renderproto 0.11.1 h14c3975_1002 conda-forge
xorg-xextproto 7.3.0 h14c3975_1002 conda-forge
xorg-xproto 7.0.31 h7f98852_1007 conda-forge
xz 5.2.5 h516909a_1 conda-forge
yaml 0.2.5 h516909a_0 conda-forge
yapf 0.30.0 pyh9f0ad1d_0 conda-forge
zeromq 4.3.3 h58526e2_3 conda-forge
zipp 3.4.0 alias xclip="xclip -selection c" py_0 conda-forge
zlib 1.2.11 h516909a_1010 conda-forge
zstd 1.4.8 ha95c52a_1 conda-forge

Minimally Reproducible Test Case

I am working on a program that needs to use another dept.'s dotnet core dll's and it works swimmingly when executed normally.

For example, considering this code:

tasks.py:

import os
import sys
sys.path.append("./dll_lib")
import clr
clr.AddReference("System.IO")
import System.IO

from celery import Celery
app = Celery('Test', broker='amqp://guest:guest@localhost:5672//', backend="rpc://")

@app.task
def test(json_file):
json_content = System.IO.File.ReadAllText(json_file)
return 1

start.sh:

#!/bin/bash

CELERY_ALWAYS_EAGER=True
CELERYD_MAX_TASKS_PER_CHILD=1
CELERYD_CONCURRENCY=1
CELERYD_PREFETCH_MULTIPLIER=1
CELERY_RESULT_BACKEND="rpc://"
CELERY_RESULT_PERSISTENT=False

celery -A tasks worker --loglevel=INFO --max-tasks-per-child=1

When I test this task it runs fine as a standalone function.

But this problem is nebulous, because I can issue a task with the path to a small json (<1mb) and it succeeds:

$ ipython -i
Python 3.7.4 (default, Aug 13 2019, 20:35:49)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.19.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import tasks

In [2]: res = tasks.test.delay("/path/small.json")

In [3]: res.ready()
Out[3]: True

But when I do this with the path to a bigger json (~7mb), it stalls indefinitely:

In [4]: res_big = tasks.test.delay("/path/bigger.json")

In [5]: res_big.ready()
Out[5]: False

Does anybody have an idea what can be related to this issue ? It looks connected to memory used.

NOTE:

This problem is not IO related, as I have tested it by reading a json as a string in the python way and using Newtonsoft.Json.dll to deserialize it. This test behaves along the similiarly with the same files, works with the small json, stalls on the bigger one.

Expected Behavior

I expect a task to succeed independent of the size of the supplied json.

Actual Behavior

Sizeable (<1 mB) json's succeed, but bigger (~ 7 mB) json stall; the task is received, but never succeeds.

Issue 2:
Checklist

I have verified that the issue exists against the master branch of Celery.

This has already been asked to the discussion group first.

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

None

Possible Duplicates

None

Environment & Settings

Celery version:

celery report Output:

celery==4.4.0
django-redis==4.10.0
redis==3.3.11

Steps to Reproduce

Required Dependencies

Minimal Python Version: 3.6

Minimal Celery Version: 4.4.0

Minimal Kombu Version: 4.6.7

Minimal Broker Version: rabbitmq

Minimal Result Backend Version: redis

Minimal OS and/or Kernel Version: ubuntu 18.04

Minimal Broker Client Version: N/A or Unknown

Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

celery==4.4.0
kombu==4.6.7
redis==3.3.11

Other Dependencies

N/A

Minimally Reproducible Test Case

Actual Behavior

I have the following code that run as expected on a single worker / code base :

@task()
def fail(*args, **kwargs):
raise NotImplementedError

@task()
def set_error(*args, **kwargs):
print('im in')

fail.s().on_error(set_error.s()).delay('foo', foo='bar').get()

As expected, behavior is:

fail task is called and raise a error

set_error is called

client gets a NotImplementedError exception

Now let say fail is on another worker. code becomes:

signature('proj.fail', queue='some-queue').on_error(set_error.s()).delay('foo', foo='bar').get()

Behavior:

fail task is called and raise an error. kombu tries to serialize the error leading to another exception (see Traceback below)

Traceback (most recent call last):
File "/.pyenv/versions/3.6.6/envs/my-env/lib/python3.6/site-packages/billiard/pool.py", line 1791, in safe_apply_callback
fun(*args, **kwargs)
File "/.pyenv/versions/3.6.6/envs/my-env/lib/python3.6/site-packages/celery/worker/request.py", line 528, in on_failure
store_result=self.store_errors,
File "/.pyenv/versions/3.6.6/envs/my-env/lib/python3.6/site-packages/celery/backends/base.py", line 165, in mark_as_failure
self._call_task_errbacks(request, exc, traceback)
File "/.pyenv/versions/3.6.6/envs/my-env/lib/python3.6/site-packages/celery/backends/base.py", line 188, in _call_task_errbacks
errback(request, exc, traceback)
File "/.pyenv/versions/3.6.6/envs/my-env/lib/python3.6/site-packages/celery/canvas.py", line 613, in __call__
return self.apply_async(args, kwargs)
File "/.pyenv/versions/3.6.6/envs/my-env/lib/python3.6/site-packages/celery/canvas.py", line 642, in apply_async
dict(self.options, **options) if options else self.options))
File "/.pyenv/versions/3.6.6/envs/my-env/lib/python3.6/site-packages/celery/canvas.py", line 669, in run
first_task.apply_async(**options)
File "/.pyenv/versions/3.6.6/envs/my-env/lib/python3.6/site-packages/celery/canvas.py", line 235, in apply_async
return _apply(args, kwargs, **options)
File "/.pyenv/versions/3.6.6/envs/my-env/lib/python3.6/site-packages/celery/app/base.py", line 771, in send_task
amqp.send_task_message(P, name, message, **options)
File "/.pyenv/versions/3.6.6/envs/my-env/lib/python3.6/site-packages/celery/app/amqp.py", line 559, in send_task_message
**properties
File "/.pyenv/versions/3.6.6/envs/my-env/lib/python3.6/site-packages/kombu/messaging.py", line 169, in publish
compression, headers)
File "/.pyenv/versions/3.6.6/envs/my-env/lib/python3.6/site-packages/kombu/messaging.py", line 252, in _prepare
body) = dumps(body, serializer=serializer)
File "/.pyenv/versions/3.6.6/envs/my-env/lib/python3.6/site-packages/kombu/serialization.py", line 221, in dumps
payload = encoder(data)
File "/.pyenv/versions/3.6.6/lib/python3.6/contextlib.py", line 99, in __exit__
self.gen.throw(type, value, traceback)
File "/.pyenv/versions/3.6.6/envs/my-env/lib/python3.6/site-packages/kombu/serialization.py", line 54, in _reraise_errors
reraise(wrapper, wrapper(exc), sys.exc_info()[2])
File "/.pyenv/versions/3.6.6/envs/my-env/lib/python3.6/site-packages/vine/five.py", line 194, in reraise
raise value.with_traceback(tb)
File "/.pyenv/versions/3.6.6/envs/my-env/lib/python3.6/site-packages/kombu/serialization.py", line 50, in _reraise_errors
yield
File "/.pyenv/versions/3.6.6/envs/my-env/lib/python3.6/site-packages/kombu/serialization.py", line 221, in dumps
payload = encoder(data)
File "/.pyenv/versions/3.6.6/envs/my-env/lib/python3.6/site-packages/kombu/utils/json.py", line 70, in dumps
**dict(default_kwargs, **kwargs))
File "/.pyenv/versions/3.6.6/lib/python3.6/json/__init__.py", line 238, in dumps
**kw).encode(obj)
File "/.pyenv/versions/3.6.6/lib/python3.6/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/.pyenv/versions/3.6.6/lib/python3.6/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "/.pyenv/versions/3.6.6/envs/my-env/lib/python3.6/site-packages/kombu/utils/json.py", line 59, in default
return super(JSONEncoder, self).default(o)
File "/.pyenv/versions/3.6.6/lib/python3.6/json/encoder.py", line 180, in default
o.__class__.__name__)
kombu.exceptions.EncodeError: Object of type 'Context' is not JSON serializable

set_error is never called

client wait forever

Expected Behavior

Context object should be serialized and error propagated.

Workaround

using pickle as default serializer fix the issue

Best regards

Nicolas