Issue 1:
Checklist

I have verified that the issue exists against the master branch of Celery.

This has already been asked to the discussion group first.

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

None

Possible Duplicates

None

Environment & Settings

Celery version:

celery report Output:

software -> celery:5.1.2 (sun-harmonics) kombu:5.1.0 py:3.8.7
billiard:3.6.4.0 py-amqp:5.0.6
platform -> system:Darwin arch:64bit
kernel version:19.6.0 imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:amqp results:rpc:///

deprecated_settings: None
broker_url: 'amqp://****:********@localhost:5672//'
result_backend: 'rpc:///'

Steps to Reproduce

Required Dependencies

Minimal Python Version: N/A or Unknown

Minimal Celery Version: 5.1.0

Minimal Kombu Version: N/A or Unknown

Minimal Broker Version: N/A or Unknown

Minimal Result Backend Version: N/A or Unknown

Minimal OS and/or Kernel Version: N/A or Unknown

Minimal Broker Client Version: N/A or Unknown

Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

amqp==5.0.6
billiard==3.6.4.0
celery==5.1.2
click==7.1.2
click-didyoumean==0.0.3
click-plugins==1.1.1
click-repl==0.2.0
kombu==5.1.0
prompt-toolkit==3.0.19
pytz==2021.1
six==1.16.0
vine==5.0.0
wcwidth==0.2.5

Other Dependencies

N/A

Minimally Reproducible Test Case

We are chaining tasks together and then adding an errback to the chain.

The expectation is that if a task in the chain fails, then the errback will be called,

and the rest of the chain skipped. Our broker is amqp (RabbitMQ), and we are using the rpc backend.

This feature worked for 4.4.x and works in 5.0.5. Starting in 5.1.0, the Celery

worker will throw an internal exception and the errback is not called. Below is a simple test case:

tasks.py

from celery import Celery
from celery.utils.log import get_task_logger

app = Celery()
app.conf.broker_url = "amqp://****:****@localhost:5672/"
app.conf.result_backend = "rpc://"

logger = get_task_logger(__name__)

@app.task
def task1():
logger.info("TASK1")
raise ValueError('foo')

@app.task
def task2():
logger.info("TASK2")

@app.task
def error_handler(request, exc, traceback):
logger.error("ERROR HANDLER")
logger.error('Task {0} raised exception: {1!r}'.format(
request.id, exc))

Then run task1 and task2 in a chain using error_handler as the errback:

from tasks import error_handler, task1, task2

chain = (task1.s() | task2.s())
x = chain.apply_async(link_error=error_handler.s())

Expected Behavior

task1 will fail and error_handler should be called:

[INFO/MainProcess] Task tasks.task1[35e2f20e-fc5b-4889-8ae0-9c1a593a15ec] received
[INFO/ForkPoolWorker-7] tasks.task1[35e2f20e-fc5b-4889-8ae0-9c1a593a15ec]: TASK1
[ERROR/ForkPoolWorker-7] tasks.error_handler[None]: ERROR HANDLER
[ERROR/ForkPoolWorker-7] tasks.error_handler[None]: Task 35e2f20e-fc5b-4889-8ae0-9c1a593a15ec raised exception: ValueError('foo')
[ERROR/ForkPoolWorker-7] Task tasks.task1[35e2f20e-fc5b-4889-8ae0-9c1a593a15ec] raised unexpected: ValueError('foo')

Actual Behavior

The Celery worker logs the following stack trace and error_handler is never called:

[ERROR/MainProcess] Pool callback raised exception: AttributeError("'dict' object has no attribute 'reply_to'")
Traceback (most recent call last):
File "/****/share/virtualenvs/celery5-reply-to-reL48Jin/lib/python3.8/site-packages/billiard/pool.py", line 1796, in safe_apply_callback
fun(*args, **kwargs)
File "/****/share/virtualenvs/celery5-reply-to-reL48Jin/lib/python3.8/site-packages/celery/worker/request.py", line 571, in on_failure
self.task.backend.mark_as_failure(
File "/****/share/virtualenvs/celery5-reply-to-reL48Jin/lib/python3.8/site-packages/celery/backends/base.py", line 199, in mark_as_failure
self.store_result(
File "/****/share/virtualenvs/celery5-reply-to-reL48Jin/lib/python3.8/site-packages/celery/backends/rpc.py", line 198, in store_result
routing_key, correlation_id = self.destination_for(task_id, request)
File "/****/share/virtualenvs/celery5-reply-to-reL48Jin/lib/python3.8/site-packages/celery/backends/rpc.py", line 179, in destination_for
return request.reply_to, request.correlation_id or task_id
AttributeError: 'dict' object has no attribute 'reply_to'

Issue 2:
PR #5423 (to address issue #2700) made a change which installs a SIGTERM handler within Task.__call__(), which effectively ignores SIGTERM. This is...er...wrong, and leads to the following problems:

Tasks cannot be called inline within a thread (eg common configurations of uwsgi, or even Django's runserver)

The warm/cold shutdown distinction is erased

The REMAP_SIGTERM configuration value is ignored (used in eg Heroku deployments)

Signal handling by non-worker calling processes is forcibly overridden

Conflicts with Celery's own worker signal handling (see worker.py and platforms.py)

The PR in question was reverted in #5577, then un-reverted in #5586.

We've had to add a custom task class and override __call__() for production use to avoid this issue. Please re-revert #5423; if #2700 is still an issue inherent to Celery (as opposed to simply warm/cold shutdown differences), then a different fix should be considered.

Edit: after some deeper looks at the code, it seems as though this only applies for inline tasks under the base Task class, as __call__() is optimized away (see trace.py) for async/dispatched tasks if it hasn't been overridden. AFAICT it would still be a problem if using a custom task class which overrides __call__() and calls super(), however.

Checklist

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Related Issues and Possible Duplicates

Related Issues

#2700

PR #5423

Possible Duplicates

None

Environment & Settings

Celery version:

celery report Output:

software -> celery:4.4.0rc3 (cliffs) kombu:4.6.4 py:3.7.4
billiard:3.6.1.0 redis:3.3.7
platform -> system:Linux arch:64bit
kernel version:4.9.184-linuxkit imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:redis results:redis://mps-redis:6379/0

Steps to Reproduce

Required Dependencies

Minimal Python Version: N/A or Unknown

Minimal Celery Version: 4.4.0rc2

Minimal Kombu Version: N/A or Unknown

Minimal Broker Version: N/A or Unknown

Minimal Result Backend Version: N/A or Unknown

Minimal OS and/or Kernel Version: N/A or Unknown

Minimal Broker Client Version: N/A or Unknown

Minimal Result Backend Client Version: N/A or Unknown

Minimally Reproducible Test Case

See https://github.com/celery/celery/blob/master/celery/app/task.py#L396

Expected Behavior

Should be able to call a task inline in a thread.

Actual Behavior

When calling task inline under Django runserver (which uses threads):

Traceback (most recent call last):
File "/app/pipelines/serializers.py", line 184, in create
remote_worker=validated_data['with_remote'],
File "/usr/local/lib/python3.7/site-packages/celery/local.py", line 191, in __call__
return self._get_current_object()(*a, **kw)
File "/usr/local/lib/python3.7/site-packages/celery/app/task.py", line 396, in __call__
signal.signal(signal.SIGTERM, handle_sigterm)
File "/usr/local/lib/python3.7/signal.py", line 47, in signal
handler = _signal.signal(_enum_to_int(signalnum), _enum_to_int(handler))
ValueError: signal only works in main thread