Issue 1:
Checklist

I have verified that the issue exists against the master branch of Celery.

This has already been asked to the discussion group first.

I have read the relevant section in the

contribution guide

on reporting bugs.

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues

in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

I have included the output of celery -A proj report in the issue.

(if you are not able to do this, then at least specify the Celery

version affected).

I have verified that the issue exists against the master branch of Celery.

I have included the contents of pip freeze in the issue.

I have included all the versions of all the external dependencies required

to reproduce this bug.

Optional Debugging Information

I have tried reproducing the issue on more than one Python version

and/or implementation.

I have tried reproducing the issue on more than one message broker and/or

result backend.

I have tried reproducing the issue on more than one version of the message

broker and/or result backend.

I have tried reproducing the issue on more than one operating system.

I have tried reproducing the issue on more than one workers pool.

I have tried reproducing the issue with autoscaling, retries,

ETA/Countdown & rate limits disabled.

I have tried reproducing the issue after downgrading

and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

None

Possible Duplicates

None

Environment & Settings

5.1.0 (sun-harmonics):

celery report Output:

software -> celery:5.1.0 (sun-harmonics) kombu:5.1.0 py:3.8.10
billiard:3.6.4.0 py-amqp:5.0.6
platform -> system:Linux arch:64bit, ELF
kernel version:5.4.117-58.216.amzn2.x86_64 imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:amqps results:redis://ter84av5ralzps6.dbif12.ng.0001.use1.cache.amazonaws.com:6379/0

broker_url: 'amqps://****:********@b-1491211e-5f2b-4f67-aaab-06509b3c7b0c.mq.us-east-1.amazonaws.com:5671//'
result_backend: 'redis://ter84av5ralzps6.dbif12.ng.0001.use1.cache.amazonaws.com:6379/0'
deprecated_settings: None
enable_utc: True
timezone: 'UTC'
task_acks_late: True
task_reject_on_worker_lost: True
worker_proc_alive_timeout: 20
worker_lost_wait: 10
worker_prefetch_multiplier: 1
task_track_started: False
result_expires: datetime.timedelta(seconds=21600)
broker_pool_limit: 8
task_routes: {
'task.http_task_executor': {'queue': 'task_work_queue'},
'task.kube_task_executor': {'queue': 'task_work_queue'},
'task.post_new': {'queue': 'task_push_queue'},
'task.post_reply': {'queue': 'task_reply_queue'},
'task.post_status_log': {'queue': 'task_status_queue'},
'task.post_status_update': {'queue': 'task_status_queue'},
'task.routing_task_transform_executor': {'queue': 'task_work_queue'}}

root@task-keeper-api-executor-worker-558785df4f-n47d5:/app#

Steps to Reproduce

Required Dependencies

Minimal Python Version: 3.8

Minimal Celery Version: 5.1.0

Minimal Kombu Version: N/A or Unknown

Minimal Broker Version: N/A or Unknown

Minimal Result Backend Version: N/A or Unknown

Minimal OS and/or Kernel Version: N/A or Unknown

Minimal Broker Client Version: N/A or Unknown

Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

root@task-keeper-api-executor-worker-558785df4f-n47d5:/app# pip freeze
aiocache @ file:///root/.cache/pypoetry/artifacts/8a/75/32/3e4230ac935365d261ded62e9b0fac56c0417eea0826c6fa9a240514d4/aiocache-0.11.1-py2.py3-none-any.whl
aiohttp @ file:///root/.cache/pypoetry/artifacts/e6/b7/73/8d17934f488836015688d06d1e12d8c3c32c493c1a1d67e6c67fc9f91c/aiohttp-3.7.4.post0-cp38-cp38-manylinux2014_x86_64.whl
alembic @ file:///root/.cache/pypoetry/artifacts/2a/a4/b0/d666d758e2d1a844c1082dadca27c5c6499bd5950b1ccbfeb0a92596bf/alembic-1.6.5-py2.py3-none-any.whl
amqp @ file:///root/.cache/pypoetry/artifacts/ed/fe/6d/d70241f03a4a3347d64443cd180317a586fdb0ce2e53dd8d0dcd06ae87/amqp-5.0.6-py3-none-any.whl
appdirs==1.4.4
asgiref @ file:///root/.cache/pypoetry/artifacts/e5/46/15/2189c1ac010f2a59bb68d74d8e7c1a948235462e517cf1cb79623a38d7/asgiref-3.3.4-py3-none-any.whl
async-timeout @ file:///root/.cache/pypoetry/artifacts/0d/5d/3e/630122e534c1b25e36c3142597c4b0b2e9d3f2e0a9cea9f10ac219f9a7/async_timeout-3.0.1-py3-none-any.whl
asyncpg @ file:///root/.cache/pypoetry/artifacts/ab/07/90/459222bf230c60a9e67b9031c827cbb87ce0bd5a4f8bdd9ead22006935/asyncpg-0.22.0-cp38-cp38-manylinux1_x86_64.whl
asyncpgsa @ file:///root/.cache/pypoetry/artifacts/6e/85/80/7c458492381c67ffd5123908b8552fccdd1bcf49fefb3b2775bc87f5bf/asyncpgsa-0.27.1.tar.gz
attrs @ file:///root/.cache/pypoetry/artifacts/1a/aa/39/10d6d07084f186f8cf6963cb033440402ad5088bb94d712239170f2ef6/attrs-21.2.0-py2.py3-none-any.whl
awscli @ file:///root/.cache/pypoetry/artifacts/39/0e/5b/37799e47334247e79fe060ea89f43b5cd1647411adb10f927d96ebe2a7/awscli-1.19.94-py2.py3-none-any.whl
backoff @ file:///root/.cache/pypoetry/artifacts/07/ec/f6/14fe7212d91fcec696718f63bd1816ebd8646466b7d9774f5e4c011a09/backoff-1.10.0-py2.py3-none-any.whl
billiard @ file:///root/.cache/pypoetry/artifacts/37/f6/65/d35c92d728dbe48ad715603358c7fae3694aca4341b9330f35eeb2c3cb/billiard-3.6.4.0-py3-none-any.whl
boto3 @ file:///root/.cache/pypoetry/artifacts/ea/81/7d/885c335fbea444ca9b34625cf0cc04137d6cdebfd3f7f758e4522aabbb/boto3-1.17.94-py2.py3-none-any.whl
botocore @ file:///root/.cache/pypoetry/artifacts/53/63/a3/4037b86bc1ae9ec11c54988ea6d9b5765522818ce0aeb5e176442f5009/botocore-1.20.94-py2.py3-none-any.whl
CacheControl==0.12.6
cachetools @ file:///root/.cache/pypoetry/artifacts/b8/bb/dd/ab39d5344a1eb90a476cc415065f1c5e74daecaf87821e82c925ca6aaa/cachetools-4.2.2-py3-none-any.whl
cachy==0.3.0
celery @ file:///root/.cache/pypoetry/artifacts/5e/9a/2b/b8daba72e801b2a1ab0d99a00f21e04a44d6fdaae5ac1fe4a6f625141f/celery-5.1.0-py3-none-any.whl
certifi==2021.5.30
cffi==1.14.5
chardet==4.0.0
cleo==0.8.1
click @ file:///root/.cache/pypoetry/artifacts/30/bc/bf/e00ffd8f0edf0294942e70e80e42a839bd2649d2c5b864e6389e526d2a/click-7.1.2-py2.py3-none-any.whl
click-didyoumean @ file:///root/.cache/pypoetry/artifacts/10/e1/17/19cd22a8318eea926ce7c9469807bbf6e887401385975a62dcb75bc419/click-didyoumean-0.0.3.tar.gz
click-plugins @ file:///root/.cache/pypoetry/artifacts/c6/6b/c8/2f62f7d37dfef3f9f445d3503ff3b3cd4d69531496e956f7363099e86f/click_plugins-1.1.1-py2.py3-none-any.whl
click-repl @ file:///root/.cache/pypoetry/artifacts/ec/dd/e3/45a9edb1cfdb658af256586577fd64af3c692acc14b32a18bb575a51c5/click_repl-0.2.0-py3-none-any.whl
clikit==0.6.2
colorama @ file:///root/.cache/pypoetry/artifacts/b7/77/29/8fe2533c9a5d409681ba402a7388c56154efbb99bf5dc1fc4c48cf821b/colorama-0.4.3-py2.py3-none-any.whl
crashtest==0.3.1
cryptography==3.4.7
distlib==0.3.2
docutils @ file:///root/.cache/pypoetry/artifacts/3d/52/66/87e71edcc3d12165b7ed9a7e33149219759e52054c101f35cfed57f5a4/docutils-0.15.2-py3-none-any.whl
fastapi @ file:///root/.cache/pypoetry/artifacts/3b/bb/c7/c7f39c78da6bc430aba8ac0b92fc5d0511f55e0e9166c45b08f0e1dc6e/fastapi-0.63.0-py3-none-any.whl
filelock==3.0.12
flake8 @ file:///root/.cache/pypoetry/artifacts/93/78/ae/ee75d1b46827f8892defc2a710979cc71803d2da75a049bdabd3adad70/flake8-3.9.2-py2.py3-none-any.whl
gunicorn @ file:///root/.cache/pypoetry/artifacts/eb/33/9c/43e158642e15413395d933fef80bb16fa6b50a6ed8dd157f506b6b2c03/gunicorn-20.1.0-py3-none-any.whl
h11 @ file:///root/.cache/pypoetry/artifacts/6f/50/9d/65e5b4940425051d03fab25052973dcf80bbe2f00676b117d40996e468/h11-0.12.0-py3-none-any.whl
html5lib==1.1
idna==2.10
iniconfig @ file:///root/.cache/pypoetry/artifacts/ac/f9/da/e990ffcd9ec361a68676a5916e391286e1ea5d1b8907ae887e141a71f5/iniconfig-1.1.1-py2.py3-none-any.whl
jeepney==0.6.0
Jinja2 @ file:///root/.cache/pypoetry/artifacts/1b/1c/64/6fcc5a7a4ffb6e923a1309d4e4d62309f637ac52cba8614c5778bc2650/Jinja2-3.0.1-py3-none-any.whl
jmespath @ file:///root/.cache/pypoetry/artifacts/43/d5/a2/f83573231324de7f5b61f5c607fbbe82ca535359a452de4852d2e25e8d/jmespath-0.10.0-py2.py3-none-any.whl
keyring==21.8.0
kombu @ file:///root/.cache/pypoetry/artifacts/62/75/03/0f5b0f86c7d2cfe40c38ca2d39d616bb71163d8848bdddcdaa0f86fc20/kombu-5.1.0-py3-none-any.whl
lockfile==0.12.2
Mako @ file:///root/.cache/pypoetry/artifacts/2b/e8/50/1d2ac0c7a28dfc4ef81a559bcb6d5dba8682c0d28edfdd91917d23a199/Mako-1.1.4-py2.py3-none-any.whl
MarkupSafe @ file:///root/.cache/pypoetry/artifacts/e9/aa/ce/73da3aa2f2d872e5d1cb28a49f7e6e0e76c9e7ba290b034f442badefd2/MarkupSafe-2.0.1-cp38-cp38-manylinux2010_x86_64.whl
mccabe @ file:///root/.cache/pypoetry/artifacts/37/6e/69/4a33a4d6c80c775b1ee205face2c6e07b762c8602bb0f0d236ebe790c5/mccabe-0.6.1-py2.py3-none-any.whl
msgpack==1.0.2
multidict @ file:///root/.cache/pypoetry/artifacts/36/92/90/7beba42c3a6efbfdc30dc864c33286fa23817b3942b21ccaf319f18ed3/multidict-5.1.0-cp38-cp38-manylinux2014_x86_64.whl
orjson @ file:///root/.cache/pypoetry/artifacts/72/11/13/e2e7fa1425adc13fa055e81e0ad53ee39758f5b27f40f139c86fe6b7fa/orjson-3.5.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
packaging==20.9
pastel==0.2.1
pexpect==4.8.0
pkginfo==1.7.0
pluggy @ file:///root/.cache/pypoetry/artifacts/9c/e5/0b/2d64d03361a081edeb5d2ec5f286ccf9719587781fbf6822e1b6384c27/pluggy-0.13.1-py2.py3-none-any.whl
poetry==1.1.6
poetry-core==1.0.3
prompt-toolkit @ file:///root/.cache/pypoetry/artifacts/b9/d1/38/231fc80979af8084d0746ac9e5bcaceb6ae7412ed7c1c1c537659d117f/prompt_toolkit-3.0.18-py3-none-any.whl
psycopg2-binary @ file:///root/.cache/pypoetry/artifacts/c6/89/2e/f0a2bb1c73cbdda3697229e4891d4fce14add7bb9b2f75dec7828e1e6f/psycopg2_binary-2.8.6-cp38-cp38-manylinux1_x86_64.whl
ptyprocess==0.7.0
py @ file:///root/.cache/pypoetry/artifacts/6b/b2/2b/e6686e7d0183dbd36bd66921efa3e77ce26260a3671524cd86614290e0/py-1.10.0-py2.py3-none-any.whl
pyasn1 @ file:///root/.cache/pypoetry/artifacts/a6/f7/5e/59a43ec23ade0888631b3f24244da7f5d5a0b6b40849c86b8c6b4c54d1/pyasn1-0.4.8-py2.py3-none-any.whl
pycodestyle @ file:///root/.cache/pypoetry/artifacts/ea/27/4a/ce8e18f033aae28e47dc2895901dce76e10e7c9efc48bcd95ab4443c47/pycodestyle-2.7.0-py2.py3-none-any.whl
pycparser==2.20
pydantic @ file:///root/.cache/pypoetry/artifacts/97/5a/97/6d56f653b6eb23c25fe8bd938764b95b9ddae8d8bcb08fb129df79f752/pydantic-1.8.2-cp38-cp38-manylinux2014_x86_64.whl
pyflakes @ file:///root/.cache/pypoetry/artifacts/b0/73/41/34d4d02987e40ffa6ee0292425303a75b4178476bf134ceaf0585a9faf/pyflakes-2.3.1-py2.py3-none-any.whl
pylev==1.4.0
pyparsing==2.4.7
pytest @ file:///root/.cache/pypoetry/artifacts/8a/b9/90/4514a24414e38b20d118ef142b175d429a18dba120914e9e0dffc5fa56/pytest-6.2.4-py3-none-any.whl
pytest-aiohttp @ file:///root/.cache/pypoetry/artifacts/a8/4e/c1/891a775ef615683510673efc2895db6b62f844caa79e6a335fcc29607c/pytest_aiohttp-0.3.0-py3-none-any.whl
python-dateutil @ file:///root/.cache/pypoetry/artifacts/1e/2c/dc/0e811c2299b40168ee1da03bc13c11762ce9fa96eb867ad22280db11fc/python_dateutil-2.8.1-py2.py3-none-any.whl
python-editor @ file:///root/.cache/pypoetry/artifacts/88/e4/9a/6a3b58d3811aabb356fe1af49b223e67e18505daf666e0dca62c12f260/python_editor-1.0.4-py3-none-any.whl
pytz @ file:///root/.cache/pypoetry/artifacts/19/50/c9/f8bffb0360052b3c85b912bc4172f824bc2571e24002806fb3a428302e/pytz-2021.1-py2.py3-none-any.whl
PyYAML @ file:///root/.cache/pypoetry/artifacts/f7/d9/62/83fb890c5915cc3c7a919097a61e93e54d7a3c47832b2f91e38224359d/PyYAML-5.4.1-cp38-cp38-manylinux1_x86_64.whl
redis @ file:///root/.cache/pypoetry/artifacts/7d/10/53/5e7715691ea8dff1b633fa5813e2a864bf74e3659c469d2a23cd6aae35/redis-3.5.3-py2.py3-none-any.whl
requests==2.25.1
requests-toolbelt==0.9.1
rsa @ file:///root/.cache/pypoetry/artifacts/3a/ab/43/a67c8d818350593872a5713ca125b95f465d62eee5ba7895de1194add1/rsa-4.7.2-py3-none-any.whl
s3transfer @ file:///root/.cache/pypoetry/artifacts/d5/26/6a/727af0fce69424f6803bc321ad17a139d3b58ceb9c82eac6d0f7ffc28a/s3transfer-0.4.2-py2.py3-none-any.whl
SecretStorage==3.3.1
shellingham==1.4.0
six==1.16.0
SQLAlchemy @ file:///root/.cache/pypoetry/artifacts/87/89/61/bd5c84081c9a69a763f109bd41e59f0f19183646d58ef4b9d48595d02d/SQLAlchemy-1.3.24-cp38-cp38-manylinux2010_x86_64.whl
starlette @ file:///root/.cache/pypoetry/artifacts/64/44/19/ddfa154bd917148cb72d6cb3d86a16bc3fa7c2c7cc7786898aee24c8b0/starlette-0.13.6-py3-none-any.whl
toml @ file:///root/.cache/pypoetry/artifacts/9f/8c/c3/7b4f6778d60f3c9fa11f8fd0e48243bbad25a04975e0a01006b6350594/toml-0.10.2-py2.py3-none-any.whl
tomlkit==0.7.2
typing-extensions @ file:///root/.cache/pypoetry/artifacts/fe/31/ef/8176395be3e4536aa430a84a1af3ade772047a9017c78ce2d045c1636b/typing_extensions-3.10.0.0-py3-none-any.whl
ujson @ file:///root/.cache/pypoetry/artifacts/bb/13/77/5f095aaa3083302d1041012b0d43375c60eb4b929d09f806fa770d4d2d/ujson-1.35.tar.gz
urllib3==1.26.5
uvicorn @ file:///root/.cache/pypoetry/artifacts/1d/0a/24/4866f58eb94ae161513ad8eb253f426f177c3695de6305af006853f27d/uvicorn-0.13.4-py3-none-any.whl
vine @ file:///root/.cache/pypoetry/artifacts/a7/a3/26/0b4cc5bb6d8828de4b6043228c8fdff0eea9acf787d1ca87cd308560a6/vine-5.0.0-py2.py3-none-any.whl
virtualenv==20.4.7
wcwidth @ file:///root/.cache/pypoetry/artifacts/36/68/e2/7232f431072d5e8aeec124120b9a1d095d45da10311d271fac10982473/wcwidth-0.2.5-py2.py3-none-any.whl
webencodings==0.5.1
yarl @ file:///root/.cache/pypoetry/artifacts/ba/ae/48/6e2cb824dd72016fe49814c19c62914323a44b9b9d3d10b583b013dbf3/yarl-1.6.3-cp38-cp38-manylinux2014_x86_64.whl
root@task-keeper-api-executor-worker-558785df4f-n47d5:/app#

Other Dependencies

ReplicationGroup:
Type: AWS::ElastiCache::ReplicationGroup
Properties:
AutomaticFailoverEnabled: true
CacheNodeType: cache.t3.medium
Engine: redis
EngineVersion: default.redis6.x
NumCacheClusters: 2

Minimally Reproducible Test Case

pipelines = []

for task in Factory.gen_tasks(request):

pipelines.append(
chain(
post_new.si(task),
http_task_executor.si(task),
group(
chain(
post_status_update.s(task.id),
post_status_log.s(task.id),
),
post_reply.s(task.id),
),
)
)

async_task = group(pipelines).apply_async()

post_new - returns None

http_task_executor - returns arg for next group

post_status_update - consumes arg from http_task_executor task

post_reply - consumes arg from http_task_executor task

Expected Behavior

Execution continue step by step

Actual Behavior

Found strange behavior implementing tasks pipeline on Celery. Most of the time tasks chain executed, but sometimes it just silently stops in the middle after the successful run of the previous task.

So most of the time (99%, probably) everything is fine and task executed after another task. But sometimes execution stops after post_new or after http_task_executor tasks

In both cases, I see in logs that the previous task completed, but the next step task is not received by a worker. Also, the queue is empty. So I started to think that a message was not sent to the broker. In debug logs, I see only information regarding internal heartbeat, no errors, no exceptions.

Also, with docker-compose this works much more stable.

Last message for the missed task execution:

[2021-06-15 19:52:32,995: INFO/ForkPoolWorker-1] Task task.http_task_executor[07cd4913-2222-4ed6-8749-ae8a663c2aec] succeeded in 1.5408981110085733s:

So, Celery reported success but no more tasks started.

Additional info:

Pool: prefork

Concurrency: 8

According to the pipeline can this happen if for some reason a worker can't store results in Redis broker and just silently ignores errors?

Another possibility is loss of messages in RabbitMQ which I will try to check

Issue 2:
Checklist

I have checked the issues list

for similar or identical enhancement to an existing feature.

I have checked the pull requests list

for existing proposed enhancements.

I have checked the commit log

to find out if the if the same enhancement was already implemented in the

master branch.

I have included all related issues and possible duplicate issues in this issue

(If there are none, check this box anyway).

Related Issues and Possible Duplicates

Related Issues

#5106 feels like it may be fixed by this as well, but I'm not sure that it fixes their exact issue

#5598 mentions that the docs don't describe this redelivered behavior

Possible Duplicates

None

Brief Summary

If task_acks_late and task_reject_on_worker_lost are set, a task will be requeued the first time it fails due to losing a worker. However if the message has the redelivered flag set to True, requeue will be set to false. That means a second worker lost failure doesn't get retried.

We're running on Heroku and it's possible we'll get workers killed back to back if we release new code and then do a config change right after, as one example. We'd like to continue to requeue the task regardless of the redelivered flag.

Design

Architectural Considerations

None

Proposed Behavior

Add a configuration flag task_requeue_on_redelivered that if set to True, causes the task to be requeued regardless of the redelivered flag.

I'm happy to put together a PR to do this if it seems like a reasonable approach.

Proposed UI/UX

Diagrams

N/A

Alternatives

The way we work around this now is by overriding Request and always setting redelivered to False:

class NeverRedeliveredRequest(Request):
def __init__(self, message, **kwargs):
if message.delivery_info:
message.delivery_info['redelivered'] = False
super().__init__(message, **kwargs)

I don't like obliterating information that might be used elsewhere though, so I'd prefer this to be in Celery itself.