Issue 1:
Checklist

I have checked the issues list

for similar or identical enhancement to an existing feature.

I have checked the pull requests list

for existing proposed enhancements.

I have checked the commit log

to find out if the if the same enhancement was already implemented in the

master branch.

I have included all related issues and possible duplicate issues in this issue

(If there are none, check this box anyway).

Related Issues and Possible Duplicates

Related Issues

#6348

The situation is similar, but this issue is based on Redis Sentinel and we use AWS ElastiCache. Error message is also a bit different.

Possible Duplicates

None

Brief Summary

When AWS ElastiCache for Redis has a replication structure and causes a failover, Celery doesn't failover and the main process tries graceful shutdown. It means,

main process exits due to Unrecoverable error when there is no remaining task

main process hangs until the remaining task finishes

The second one is critical for us because we have many long ongoing tasks. The first one is better than it because we automatically reboot the process, but we'd like to avoid it because Redis failover is a kind of expected behavior.

If some change is acceptable, we will try to create a pull request by ourselves.

Detail

We got the following error when ElastiCache failover happened.

[CRITICAL - 2022-11-04 10:45:27 - 21 worker:207] Unrecoverable error: ReadOnlyError("You can't write against a read only replica.")
Traceback (most recent call last):
File "/var/www/venv/lib/python3.7/site-packages/celery/worker/worker.py", line 203, in start
self.blueprint.start(self)
File "/var/www/venv/lib/python3.7/site-packages/celery/bootsteps.py", line 116, in start
step.start(parent)
File "/var/www/venv/lib/python3.7/site-packages/celery/bootsteps.py", line 365, in start
return self.obj.start()
File "/var/www/venv/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 326, in start
blueprint.start(self)
File "/var/www/venv/lib/python3.7/site-packages/celery/bootsteps.py", line 116, in start
step.start(parent)
File "/var/www/venv/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 618, in start
c.loop(*c.loop_args())
File "/var/www/venv/lib/python3.7/site-packages/celery/worker/loops.py", line 97, in asynloop
next(loop)
File "/var/www/venv/lib/python3.7/site-packages/kombu/asynchronous/hub.py", line 362, in create_loop
cb(*cbargs)
File "/var/www/venv/lib/python3.7/site-packages/kombu/transport/redis.py", line 1266, in on_readable
self.cycle.on_readable(fileno)
File "/var/www/venv/lib/python3.7/site-packages/kombu/transport/redis.py", line 504, in on_readable
chan.handlers[type]()
File "/var/www/venv/lib/python3.7/site-packages/kombu/transport/redis.py", line 898, in _brpop_read
**options)
File "/var/www/venv/lib/python3.7/site-packages/redis/client.py", line 1189, in parse_response
response = connection.read_response()
File "/var/www/venv/lib/python3.7/site-packages/redis/connection.py", line 817, in read_response
raise response
redis.exceptions.ReadOnlyError: You can't write against a read only replica.

As far as we checked, an error during connection becomes ConnectionError and regarded as "Recoverable", but ReadOnlyError (and its parent ResponseError) is regarded as "Unrecoverable".

Design

Architectural Considerations

Maybe this part in kombu is related

https://github.com/celery/kombu/blob/v5.2.3/kombu/transport/redis.py#L128-L141

Proposed Behavior

When an error happens due to switching master to replica, retry connection and connect to the new master.

Proposed UI/UX

Adding some option to regard ResponseError as "Recoverable". Configuration key like redis_retry_on_response_error or redis_retry_on_failover?

But we're not sure if regarding ResponseError as "Recoverable" has a critical impact on existing behavior. We'd really appreciate advice from the community.

Diagrams

N/A

Alternatives

None

Issue 2:
Checklist

I have checked the issues list

for similar or identical bug reports.

I have checked the pull requests list

for existing proposed fixes.

I have checked the commit log

to find out if the bug was already fixed in the master branch.

I have included all related issues and possible duplicate issues in this issue

(If there are none, check this box anyway).

Related Issues and Possible Duplicates

Related Issues

None

Possible Duplicates

None

Description

The migration command stated in the documentation doesn't work. The following works instead:

python manage.py migrate django_celery_results

Suggestions