Issue 1:
Checklist

I have verified that the issue exists against the master branch of Celery. (not against latest commit specifically but against latest version 4.4.5)

This has already been asked to the discussion group first.

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

None

Possible Duplicates

None

Environment & Settings

Celery version: 4.4.5 (cliffs)

Celery Report:

https://pastebin.com/0x13Xjiq

Steps to Reproduce

Required Dependencies

Minimal Python Version: 3.6.8

Minimal Celery Version: 4.4.1

Minimal Kombu Version: 4.6.8

Minimal Broker Version: Unknown

Minimal Result Backend Version: Unknown

Minimal OS and/or Kernel Version: Ubuntu 18

Minimal Broker Client Version: Unknown

Minimal Result Backend Client Version: Unknown

Python Packages

pip freeze Output:

Only relevant packages shown below as 15five is not open-source:
amqp==2.6.0
billiard==3.6.3.0
celery==4.4.5
Django==2.2.13
django-celery-results==1.2.1
future==0.18.2
importlib-metadata==1.6.1
kombu==4.6.10
vine==1.3.0
pytz==2020.1
zipp==3.1.0

Other Dependencies

broker: AWS elasticache redis

Minimally Reproducible Test Case

https://github.com/caleb15/celeryPickleBugReproduction

Expected Behavior

Error to not happen. If error is a result of our faulty user code, then I would expect the celery queue to log a error and continue operating.

Actual Behavior

We get a AttributeError and the queue dies. Upon celery restart the queue continues to process events, but eventually hit the same error and dies again.

I can send a link to the sentry error event directly to a celery engineer if one is interested. Otherwise the relevant info is below:

AttributeError billiard.reduction in loadbuf
Error on stopping Pool: "AttributeError(\"can't set attribute\")"
billiard/reduction.py in loadbuf at line 61
billiard/connection.py in recv at line 279
celery/concurrency/asynpool.py in _flush_outqueue at line 412
celery/concurrency/asynpool.py in iterate_file_descriptors_safely at line 240
celery/concurrency/asynpool.py in on_stop_not_started at line 387
billiard/pool.py in stop at line 526
billiard/pool.py in stop_if_not_current at line 145
billiard/pool.py in join at line 1622
celery/concurrency/prefork.py in on_stop at line 145
celery/concurrency/base.py in stop at line 123
celery/bootsteps.py in stop at line 373
celery/bootsteps.py in send_all at line 151

Another stacktrace:

celery/concurrency/asynpool.py in _recv_message at line 332
celery/concurrency/asynpool.py in on_result_readable at line 351
kombu/asynchronous/hub.py in create_loop at line 364
celery/worker/loops.py in asynloop at line 83
celery/worker/consumer/consumer.py in start at line 599
celery/bootsteps.py in start at line 119
celery/worker/consumer/consumer.py in start at line 318
celery/bootsteps.py in start at line 369
celery/bootsteps.py in start at line 119
celery/worker/worker.py in start at line 208

sys.argv:

[/srv/venvs/py3/lib/python3.7/site-packages/celery/__main__.py, worker, -A, ff, --loglevel=INFO, -Q, celery, --logfile=/var/log/celery/default%I.log, --pidfile=/var/run/celery/default.pid, --hostname=default@celery0]

What we have tried so far:

We have tried the following without success:

Updating python version from 3.6.8 to 3.7.7

Upgrading celery

Moving most tasks out of the default queue into other queues

pinpointing which task caused the error by looking through logs (we have been unable to pinpoint any specific task, the last task processed before error varies)

[update] identification & cause of error:

Our staff engineer noticed the task responsible in the sentry breadcrumbs and I figured out that the task was raising a ValidationError that I suspected was not pickleable. We ignored the error and so far the queue hasn't died on us. My suspicion was confirmed when a CI failure revealed that the error was unpickleable:

test_update_billing_add_ons_ignores_simultaneous_request_error (ff.mods.billing.tests.test_subscription_logic.UpdateBillingAddOnsTestCase) failed:

ValidationError(b'<?xml version="1.0" encoding="UTF-8"?>\n<error>\n
<symbol>simultaneous_request</symbol>\n <description>A change for
subscription 456ee8a3881160536ad68346fe91eede is already in
progress.</description>\n</error>')

Unfortunately, the exception it raised cannot be pickled, making it impossible
for the parallel test runner to handle it cleanly.

Here's the error encountered while trying to pickle the exception:

AttributeError("can't set attribute")

You should re-run this test with the --parallel=1 option to reproduce the
failure and get a correct traceback.

Destroying test database for alias 'default'...
TOTAL RUNTIME: 67.09s
TOTAL RUNTIME: 67.09s
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/local/lib/python3.7/multiprocessing/pool.py", line 121, in worker
result = (True, func(*args, **kwds))
File "/home/circleci/project/venv/lib/python3.7/site-packages/django/test/runner.py", line 309, in _run_subsuite
result = runner.run(subsuite)
File "/home/circleci/project/venv/lib/python3.7/site-packages/django/test/runner.py", line 256, in run
test(result)
File "/usr/local/lib/python3.7/unittest/suite.py", line 84, in __call__
return self.run(*args, **kwds)
File "/usr/local/lib/python3.7/unittest/suite.py", line 122, in run
test(result)
File "/home/circleci/project/venv/lib/python3.7/site-packages/django/test/testcases.py", line 271, in __call__
super().__call__(result)
File "/usr/local/lib/python3.7/unittest/case.py", line 676, in __call__
return self.run(*args, **kwds)
File "/usr/local/lib/python3.7/unittest/case.py", line 636, in run
self._feedErrorsToResult(result, outcome.errors)
File "/usr/local/lib/python3.7/unittest/case.py", line 566, in _feedErrorsToResult
result.addError(test, exc_info)
File "/home/circleci/project/venv/lib/python3.7/site-packages/django/test/runner.py", line 197, in addError
self.check_picklable(test, err)
File "/home/circleci/project/venv/lib/python3.7/site-packages/django/test/runner.py", line 130, in check_picklable
self._confirm_picklable(err)
File "/home/circleci/project/venv/lib/python3.7/site-packages/django/test/runner.py", line 104, in _confirm_picklable
pickle.loads(pickle.dumps(obj))
AttributeError: can't set attribute

Issue 2:
Checklist

I have verified that the issue exists against the master branch of Celery.

This has already been asked to the discussion group first.

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

None

Possible Duplicates

None

Environment & Settings

Celery version:

celery report Output:

software -> celery:5.2.0rc1 (dawn-chorus) kombu:5.2.0rc1 py:3.7.12
billiard:3.6.4.0 py-amqp:5.0.6
platform -> system:Linux arch:64bit, ELF
kernel version:5.10.47-linuxkit imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:pyamqp results:disabled

broker_url: 'amqp://guest:********@broker:5672//'
deprecated_settings: None

Steps to Reproduce

Required Dependencies

Minimal Python Version: 3.7

Minimal Celery Version: 5.1.2

Minimal Kombu Version: N/A

Minimal Broker Version: N/A

Minimal Result Backend Version: N/A

Minimal OS and/or Kernel Version: N/A

Minimal Broker Client Version: N/A

Minimal Result Backend Client Version: N/A

Python Packages

pip freeze Output:

amqp==5.0.6
billiard==3.6.4.0
cached-property==1.5.2
celery @ git+https://github.com/celery/celery@49452916f94d5ec60af246cea600855e6d976b48
click==8.0.1
click-didyoumean==0.3.0
click-plugins==1.1.1
click-repl==0.2.0
importlib-metadata==4.8.1
kombu==5.2.0rc1
prompt-toolkit==3.0.20
pytz==2021.3
six==1.16.0
typing-extensions==3.10.0.2
vine==5.0.0
wcwidth==0.2.5
zipp==3.6.0

Other Dependencies

N/A

Minimally Reproducible Test Case

from celery import Celery

app = Celery("tasks", broker="pyamqp://guest@broker//")

class BrokenException(Exception):
"""
This Exception can't be unpickled by default
"""
def __init__(self, msg, other_arg):
super().__init__(msg)
self.other_arg = other_arg

@app.task(bind=True)
def unsafe_exception(self):
self.retry(exc=BrokenException("msg", "other_arg"))

def main():
unsafe_exception.delay()

if __name__ == "__main__":
main()

Then run with python tasks.py

Repo here:

https://github.com/amcgavin/celery-issue/

Expected Behavior

Exception occurs in the worker handler, but handled gracefully.

note: this was the behaviour in celery==5.0.2

Actual Behavior

worker process crashes, with exit code 0.

Note that raising the exception directly doesn't crash (see the repo for detailed example).

celery_1 | [2021-10-08 08:34:10,200: INFO/MainProcess] Task tasks.unsafe_exception[38b08c18-451b-4645-a4ac-ada2e6b43e5e] received
celery_1 | [2021-10-08 08:34:10,201: CRITICAL/MainProcess] Unrecoverable error: TypeError("__init__() missing 1 required positional argument: 'other_arg'")
celery_1 | Traceback (most recent call last):
celery_1 | File "/usr/local/lib/python3.7/site-packages/celery/worker/worker.py", line 203, in start
celery_1 | self.blueprint.start(self)
celery_1 | File "/usr/local/lib/python3.7/site-packages/celery/bootsteps.py", line 116, in start
celery_1 | step.start(parent)
celery_1 | File "/usr/local/lib/python3.7/site-packages/celery/bootsteps.py", line 365, in start
celery_1 | return self.obj.start()
celery_1 | File "/usr/local/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 326, in start
celery_1 | blueprint.start(self)
celery_1 | File "/usr/local/lib/python3.7/site-packages/celery/bootsteps.py", line 116, in start
celery_1 | step.start(parent)
celery_1 | File "/usr/local/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 618, in start
celery_1 | c.loop(*c.loop_args())
celery_1 | File "/usr/local/lib/python3.7/site-packages/celery/worker/loops.py", line 81, in asynloop
celery_1 | next(loop)
celery_1 | File "/usr/local/lib/python3.7/site-packages/kombu/asynchronous/hub.py", line 361, in create_loop
celery_1 | cb(*cbargs)
celery_1 | File "/usr/local/lib/python3.7/site-packages/celery/concurrency/asynpool.py", line 325, in on_result_readable
celery_1 | next(it)
celery_1 | File "/usr/local/lib/python3.7/site-packages/celery/concurrency/asynpool.py", line 306, in _recv_message
celery_1 | message = load(bufv)
celery_1 | TypeError: __init__() missing 1 required positional argument: 'other_arg'
celery_1 | [2021-10-08 08:34:41,378: ERROR/MainProcess] Task handler raised error: WorkerLostError('Worker exited prematurely: exitcode 0 Job: 2.')
celery_1 | Traceback (most recent call last):
celery_1 | File "/usr/local/lib/python3.7/site-packages/celery/worker/worker.py", line 203, in start
celery_1 | self.blueprint.start(self)
celery_1 | File "/usr/local/lib/python3.7/site-packages/celery/bootsteps.py", line 116, in start
celery_1 | step.start(parent)
celery_1 | File "/usr/local/lib/python3.7/site-packages/celery/bootsteps.py", line 365, in start
celery_1 | return self.obj.start()
celery_1 | File "/usr/local/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 326, in start
celery_1 | blueprint.start(self)
celery_1 | File "/usr/local/lib/python3.7/site-packages/celery/bootsteps.py", line 116, in start
celery_1 | step.start(parent)
celery_1 | File "/usr/local/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 618, in start
celery_1 | c.loop(*c.loop_args())
celery_1 | File "/usr/local/lib/python3.7/site-packages/celery/worker/loops.py", line 81, in asynloop
celery_1 | next(loop)
celery_1 | File "/usr/local/lib/python3.7/site-packages/kombu/asynchronous/hub.py", line 361, in create_loop
celery_1 | cb(*cbargs)
celery_1 | File "/usr/local/lib/python3.7/site-packages/celery/concurrency/asynpool.py", line 325, in on_result_readable
celery_1 | next(it)
celery_1 | File "/usr/local/lib/python3.7/site-packages/celery/concurrency/asynpool.py", line 306, in _recv_message
celery_1 | message = load(bufv)
celery_1 | TypeError: __init__() missing 1 required positional argument: 'other_arg'
celery_1 |
celery_1 | During handling of the above exception, another exception occurred:
celery_1 |
celery_1 | Traceback (most recent call last):
celery_1 | File "/usr/local/lib/python3.7/site-packages/billiard/pool.py", line 1267, in mark_as_worker_lost
celery_1 | human_status(exitcode), job._job),
celery_1 | billiard.exceptions.WorkerLostError: Worker exited prematurely: exitcode 0 Job: 2.
celery-example_celery_1 exited with code 0