Issue 1:
Checklist

I have verified that the issue exists against the master branch of Celery.

This has already been asked to the discussions forum first.

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

None

Possible Duplicates

None

Environment & Settings

Celery version: 5.2.0 (dawn-chorus)

celery report Output:

software -> celery:5.2.0 (dawn-chorus) kombu:5.2.1 py:3.8.6
billiard:3.6.4.0 py-amqp:5.0.6
platform -> system:Darwin arch:64bit
kernel version:18.7.0 imp:CPython
loader -> celery.loaders.default.Loader
settings -> transport:amqp results:disabled

deprecated_settings: None

Steps to Reproduce

create task.py with contents:

@app.task(autoretry_for=(Exception,), retry_backoff=True, name="task", acks_late=True)
def task():
print("failing")
raise Exception()

if __name__ == '__main__':
task.apply_async(queue='task', expires=60)

run celery -A task worker -n task -Q task

run python task.py

Required Dependencies

Minimal Python Version: 3.8.6

Minimal Celery Version: 5.2.0

Minimal Kombu Version: 5.2.1

Minimal Broker Version: RabbitMQ 3.8.16

Minimal Result Backend Version: N/A or Unknown

Minimal OS and/or Kernel Version: N/A or Unknown

Minimal Broker Client Version: N/A or Unknown

Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

amqp==5.0.6
billiard==3.6.4.0
celery==5.2.0
click==8.0.3
click-didyoumean==0.3.0
click-plugins==1.1.1
click-repl==0.2.0
kombu==5.2.1
prompt-toolkit==3.0.22
pytz==2021.3
redis==3.5.3
six==1.16.0
vine==5.0.0
wcwidth==0.2.5

Other Dependencies

N/A

Minimally Reproducible Test Case

@app.task(autoretry_for=(Exception,), retry_backoff=True, name="task", acks_late=True)
def task():
print("failing")
raise Exception()

if __name__ == '__main__':
task.apply_async(queue='task', expires=60)

Expected Behavior

the task should run and fail 3 times

Actual Behavior

the task runs and fails once, then the following error

[2021-11-14 18:47:19,555: WARNING/ForkPoolWorker-2] Task task[01e23705-69eb-4021-82fa-7653611cd15f] reject requeue=False: '<' not supported between instances of 'str' and 'int'
Traceback (most recent call last):
File "/Users/pelegvia/.pyenv/versions/via-algo-maps3.8.6/lib/python3.8/site-packages/celery/app/autoretry.py", line 34, in run
return task._orig_run(*args, **kwargs)
File "/Users/pelegvia/workspace/task-manager/task.py", line 25, in task
raise Exception()
Exception

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/pelegvia/.pyenv/versions/via-algo-maps3.8.6/lib/python3.8/site-packages/celery/app/task.py", line 734, in retry
S.apply_async()
File "/Users/pelegvia/.pyenv/versions/via-algo-maps3.8.6/lib/python3.8/site-packages/celery/canvas.py", line 219, in apply_async
return _apply(args, kwargs, **options)
File "/Users/pelegvia/.pyenv/versions/via-algo-maps3.8.6/lib/python3.8/site-packages/celery/app/task.py", line 575, in apply_async
return app.send_task(
File "/Users/pelegvia/.pyenv/versions/via-algo-maps3.8.6/lib/python3.8/site-packages/celery/app/base.py", line 741, in send_task
if expires_s < 0:
TypeError: '<' not supported between instances of 'str' and 'int'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/pelegvia/.pyenv/versions/via-algo-maps3.8.6/lib/python3.8/site-packages/celery/app/trace.py", line 451, in trace_task
R = retval = fun(*args, **kwargs)
File "/Users/pelegvia/.pyenv/versions/via-algo-maps3.8.6/lib/python3.8/site-packages/celery/app/trace.py", line 734, in __protected_call__
return self.run(*args, **kwargs)
File "/Users/pelegvia/.pyenv/versions/via-algo-maps3.8.6/lib/python3.8/site-packages/celery/app/autoretry.py", line 54, in run
ret = task.retry(exc=exc, **retry_kwargs)
File "/Users/pelegvia/.pyenv/versions/via-algo-maps3.8.6/lib/python3.8/site-packages/celery/app/task.py", line 736, in retry
raise Reject(exc, requeue=False)
celery.exceptions.Reject: (TypeError("'<' not supported between instances of 'str' and 'int'"), False)

Notes

This does not happen on 5.1.2 only on 5.2.0

Issue 2:
Checklist

I have verified that the issue exists against the master branch of Celery.

This has already been asked to the discussion group first.

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Confirmed to exist on at least celery 4.3.0 as well.

Related Issues and Possible Duplicates

Related Issues

#3709 is related to error handling for chords, though this is more of an edge case in which the behavior is problematic, rather than just misalignment of docs and actual behavior.

Possible Duplicates

There are several other issues that deal with chains-in-chords, e.g. #5229 seems similar, but nothing that addresses this exact problem. I don't think this duplicates them, but I haven't dug into the celery code enough to be sure that they aren't caused the the same underlying misbehavior in the redis backend.

Environment & Settings

Celery version: 4.4.6 (cliffs)

celery report Output:

software -> celery:4.4.6 (cliffs) kombu:4.6.11 py:3.6.9
billiard:3.6.3.0 redis:3.5.3
platform -> system:Linux arch:64bit, ELF
kernel version:4.4.0-17763-Microsoft imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:redis results:redis://localhost:6379/

broker_url: 'redis://localhost:6379//'
result_backend: 'redis://localhost:6379/'

Steps to Reproduce

Required Dependencies

Not sure that any of these are minimal, but I'll include what I can test locally:

Minimal Python Version: 3.6

Minimal Celery Version: 4.3

Minimal Kombu Version: N/A or Unknown

Minimal Broker Version: N/A or Unknown

Minimal Result Backend Version: 6.0.4

Minimal OS and/or Kernel Version: N/A or Unknown

Minimal Broker Client Version: N/A or Unknown

Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

amqp==2.6.0
billiard==3.6.3.0
celery==4.4.6
future==0.18.2
importlib-metadata==1.7.0
kombu==4.6.11
pkg-resources==0.0.0
pytz==2020.1
redis==3.5.3
vine==1.3.0
zipp==3.1.0

Other Dependencies

Output of redis-server --version, in case it's useful:

Redis server v=6.0.4 sha=00000000:0 malloc=jemalloc-5.1.0 bits=64 build=d5d470262d47f1a0

Minimally Reproducible Test Case

Basic test case below. I'm still working on converting this to an integration test for your test suite.

from celery import Celery

app = Celery('task_fail', backend='redis://localhost:6379', broker='redis://localhost:6379')

@app.task
def simple_pass():
print("Passing")
pass

@app.task
def simple_fail():
print("Failing!")
raise Exception('Fail!')

@app.task
def record_failure(*args, **kwargs):
print(f'Fail with args={args}, kwargs={kwargs}')

if __name__ == "__main__":
from celery import chord

# Test 1: Passes - When the chain fails at the end, the failure logic on
# the chord is executed.
try:
chain = simple_pass.si() | simple_fail.si()
~chord([simple_pass.si(), chain],
simple_pass.si().on_error(record_failure.s(test="test_1")))
except Exception:
print("Test 1 worked")

# Test 2: Fails - When the chain has an intermediate exception, the failure
# logic on the chord is not executed & the client simply hangs.
try:
chain = simple_pass.si() | simple_fail.si() | simple_pass.si()
~chord([simple_pass.si(), chain],
simple_pass.si().on_error(record_failure.s(test="test_2")))
except Exception:
print("Test 2 worked")

Note that the only difference between test 1 and test 2 is that test 2 add the simple_pass.si() call to the end of the chain. That seems to be enough to trigger this error.

Expected Behavior

Test 2 should behave the same way that Test 1 does. Output of Test 1, with some simplification:

[2020-07-10 15:28:52,940: INFO/MainProcess] Received task: task_fail.simple_pass[65241973-d598-4bbe-a0c8-1f86694c9f33]
[2020-07-10 15:28:52,945: WARNING/ForkPoolWorker-8] Passing
[2020-07-10 15:28:52,954: INFO/ForkPoolWorker-8] Task task_fail.simple_pass[65241973-d598-4bbe-a0c8-1f86694c9f33] succeeded in 0.008173699956387281s: None
[2020-07-10 15:28:52,967: INFO/MainProcess] Received task: task_fail.simple_pass[fc15c163-0616-4b87-94c3-2c1b225f2e0e]
[2020-07-10 15:28:52,972: WARNING/ForkPoolWorker-8] Passing
[2020-07-10 15:28:52,985: INFO/ForkPoolWorker-8] Task task_fail.simple_pass[fc15c163-0616-4b87-94c3-2c1b225f2e0e] succeeded in 0.013192899990826845s: None
[2020-07-10 15:28:52,985: INFO/MainProcess] Received task: task_fail.simple_fail[0b0a3925-7ea8-4423-8e87-3a9f00d5c967]
[2020-07-10 15:28:52,992: WARNING/ForkPoolWorker-8] Failing!
[2020-07-10 15:28:53,010: ERROR/ForkPoolWorker-8] Chord callback for '76840678-80a3-4915-96b8-04c135ecb237' raised: ChordError("Dependency 0b0a3925-7ea8-4423-8e87-3a9f00d5c967 raised Exception('Fail!',)",)
Traceback (most recent call last):
File "~/celery/celery.venv/lib/python3.6/site-packages/celery/app/trace.py", line 412, in trace_task
R = retval = fun(*args, **kwargs)
File "~/celery/celery.venv/lib/python3.6/site-packages/celery/app/trace.py", line 704, in __protected_call__
return self.run(*args, **kwargs)
File "~/celery/task_fail.py", line 13, in simple_fail
raise Exception('Fail!')
Exception: Fail!

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "~/celery/celery.venv/lib/python3.6/site-packages/celery/backends/redis.py", line 449, in on_chord_part_return
callback.delay([unpack(tup, decode) for tup in resl])
File "~/celery/celery.venv/lib/python3.6/site-packages/celery/backends/redis.py", line 449, in <listcomp>
callback.delay([unpack(tup, decode) for tup in resl])
File "~/celery/celery.venv/lib/python3.6/site-packages/celery/backends/redis.py", line 402, in _unpack_chord_result
raise ChordError('Dependency {0} raised {1!r}'.format(tid, retval))
celery.exceptions.ChordError: Dependency 0b0a3925-7ea8-4423-8e87-3a9f00d5c967 raised Exception('Fail!',)
[2020-07-10 15:28:53,022: INFO/MainProcess] Received task: task_fail.record_failure[5f264f5d-a330-4f01-9274-662474d31a8f]
[2020-07-10 15:28:53,026: ERROR/ForkPoolWorker-8] Task task_fail.simple_fail[0b0a3925-7ea8-4423-8e87-3a9f00d5c967] raised unexpected: Exception('Fail!',)
Traceback (most recent call last):
File "~/celery/celery.venv/lib/python3.6/site-packages/celery/app/trace.py", line 412, in trace_task
R = retval = fun(*args, **kwargs)
File "~/celery/celery.venv/lib/python3.6/site-packages/celery/app/trace.py", line 704, in __protected_call__
return self.run(*args, **kwargs)
File "~/celery/task_fail.py", line 13, in simple_fail
raise Exception('Fail!')
Exception: Fail![2020-07-10 15:28:53,028: WARNING/ForkPoolWorker-1] Fail with args=('bc3aeb74-68c1-40e9-9c55-be9fb7860786',), kwargs={'test': 'test_1'}
[2020-07-10 15:28:53,036: INFO/ForkPoolWorker-1] Task task_fail.record_failure[5f264f5d-a330-4f01-9274-662474d31a8f] succeeded in 0.007674199994653463s: None

The most important thing to notice here is that the record_failure task is called & logs the failure, including the ID of the failing task. I expect this behavior to occur in both cases.

Actual Behavior

The output of the worker on Test 2 is:

[2020-07-10 15:28:55,039: INFO/MainProcess] Received task: task_fail.simple_pass[5779ce6a-64af-40ea-9b26-8ff3bdad348d]
[2020-07-10 15:28:55,050: INFO/MainProcess] Received task: task_fail.simple_pass[39469dfa-401c-45f0-9314-5849f6e7ca5f]
[2020-07-10 15:28:55,051: WARNING/ForkPoolWorker-8] Passing
[2020-07-10 15:28:55,051: WARNING/ForkPoolWorker-1] Passing
[2020-07-10 15:28:55,058: INFO/ForkPoolWorker-8] Task task_fail.simple_pass[5779ce6a-64af-40ea-9b26-8ff3bdad348d] succeeded in 0.007474799989722669s: None
[2020-07-10 15:28:55,062: INFO/ForkPoolWorker-1] Task task_fail.simple_pass[39469dfa-401c-45f0-9314-5849f6e7ca5f] succeeded in 0.010612400015816092s: None
[2020-07-10 15:28:55,069: INFO/MainProcess] Received task: task_fail.simple_fail[2fead196-cfe2-4ce5-896f-303c2f9970e2]
[2020-07-10 15:28:55,073: WARNING/ForkPoolWorker-8] Failing!
[2020-07-10 15:28:55,081: ERROR/ForkPoolWorker-8] Task task_fail.simple_fail[2fead196-cfe2-4ce5-896f-303c2f9970e2] raised unexpected: Exception('Fail!',)
Traceback (most recent call last):
File "/home/crawjor/test/celery/celery.venv/lib/python3.6/site-packages/celery/app/trace.py", line 412, in trace_task
R = retval = fun(*args, **kwargs)
File "/home/crawjor/test/celery/celery.venv/lib/python3.6/site-packages/celery/app/trace.py", line 704, in __protected_call__
return self.run(*args, **kwargs)
File "/home/crawjor/test/celery/task_fail.py", line 13, in simple_fail
raise Exception('Fail!')
Exception: Fail!

Then no more progress is made. Note that neither the error handler nor the callback is ever invoked, despite the chain failing (which should trigger failure handlers in the chord).

The client side also hangs waiting on redis:

File "task_fail.py", line 40, in <module>
simple_pass.si().on_error(record_failure.s(test="test_2")))
File "~/celery/celery.venv/lib/python3.6/site-packages/celery/canvas.py", line 475, in __invert__
return self.apply_async().get()
File "~/celery/celery.venv/lib/python3.6/site-packages/celery/result.py", line 237, in get
on_message=on_message,
File "~/celery/celery.venv/lib/python3.6/site-packages/celery/backends/asynchronous.py", line 200, in wait_for_pending
for _ in self._wait_for_pending(result, **kwargs):
File "~/celery/celery.venv/lib/python3.6/site-packages/celery/backends/asynchronous.py", line 268, in _wait_for_pending
on_interval=on_interval):
File "~/celery/celery.venv/lib/python3.6/site-packages/celery/backends/asynchronous.py", line 55, in drain_events_until
yield self.wait_for(p, wait, timeout=interval)
File "~/celery/celery.venv/lib/python3.6/site-packages/celery/backends/asynchronous.py", line 64, in wait_for
wait(timeout=timeout)
File "~/celery/celery.venv/lib/python3.6/site-packages/celery/backends/redis.py", line 161, in drain_events
message = self._pubsub.get_message(timeout=timeout)
File "~/celery/celery.venv/lib/python3.6/site-packages/redis/client.py", line 3617, in get_message
response = self.parse_response(block=False, timeout=timeout)
File "~/celery/celery.venv/lib/python3.6/site-packages/redis/client.py", line 3503, in parse_response
if not block and not conn.can_read(timeout=timeout):
File "~/celery/celery.venv/lib/python3.6/site-packages/redis/connection.py", line 734, in can_read
return self._parser.can_read(timeout)
File "~/celery/celery.venv/lib/python3.6/site-packages/redis/connection.py", line 321, in can_read
return self._buffer and self._buffer.can_read(timeout)
File "~/celery/celery.venv/lib/python3.6/site-packages/redis/connection.py", line 231, in can_read
raise_on_timeout=False)
File "~/celery/celery.venv/lib/python3.6/site-packages/redis/connection.py", line 198, in _read_from_socket
data = recv(self._sock, socket_read_size)
File "~/celery/celery.venv/lib/python3.6/site-packages/redis/_compat.py", line 72, in recv

There are no active, revoked, scheduled, or reserved tasks, and the queue is empty:

~/celery$ celery -A task_fail inspect active
-> celery: OK
- empty -
~/celery$ celery -A task_fail inspect scheduled
-> celery: OK
- empty -
~/celery$ celery -A task_fail inspect revoked
-> celery: OK
- empty -
~/celery$ celery -A task_fail inspect reserved
-> celery: OK
- empty -