Repository: celery/celery

Issue:
Regard backend redis's switching to replica as recoverable error
<!--
Please fill this template entirely and do not erase parts of it.
We reserve the right to close without a response
enhancement requests which are incomplete.
-->
# Checklist
<!--
To check an item on the list replace [ ] with [x].
-->

- [x] I have checked the [issues list](https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Enhancement%22+-label%3A%22Category%3A+Documentation%22)
  for similar or identical enhancement to an existing feature.
- [x] I have checked the [pull requests list](https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22Issue+Type%3A+Enhancement%22+-label%3A%22Category%3A+Documentation%22)
  for existing proposed enhancements.
- [x] I have checked the [commit log](https://github.com/celery/celery/commits/master)
  to find out if the if the same enhancement was already implemented in the
  master branch.
- [x] I have included all related issues and possible duplicate issues in this issue
      (If there are none, check this box anyway).

## Related Issues and Possible Duplicates
<!--
Please make sure to search and mention any related issues
or possible duplicates to this issue as requested by the checklist above.

This may or may not include issues in other repositories that the Celery project
maintains or other repositories that are dependencies of Celery.

If you don't know how to mention issues, please refer to Github's documentation
on the subject: https://help.github.com/en/articles/autolinked-references-and-urls#issues-and-pull-requests
-->

#### Related Issues

- https://github.com/celery/celery/issues/6348
  - The situation is similar, but this issue is based on Redis Sentinel and we use AWS ElastiCache. Error message is also a bit different.

#### Possible Duplicates

- None

# Brief Summary
<!--
Please include a brief summary of what the enhancement is
and why it is needed.
-->

When AWS ElastiCache for Redis has a replication structure and causes a failover, Celery doesn't failover and the main process tries graceful shutdown. It means,
- main process exits due to Unrecoverable error when there is no remaining task
- main process hangs until the remaining task finishes

The second one is critical for us because we have many long ongoing tasks. The first one is better than it because we automatically reboot the process, but we'd like to avoid it because Redis failover is a kind of expected behavior.

If some change is acceptable, we will try to create a pull request by ourselves.

# Detail

We got the following error when ElastiCache failover happened.
```
[CRITICAL - 2022-11-04 10:45:27 - 21 worker:207] Unrecoverable error: ReadOnlyError("You can't write against a read only replica.")
Traceback (most recent call last):
  File "/var/www/venv/lib/python3.7/site-packages/celery/worker/worker.py", line 203, in start
    self.blueprint.start(self)
  File "/var/www/venv/lib/python3.7/site-packages/celery/bootsteps.py", line 116, in start
    step.start(parent)
  File "/var/www/venv/lib/python3.7/site-packages/celery/bootsteps.py", line 365, in start
    return self.obj.start()
  File "/var/www/venv/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 326, in start
    blueprint.start(self)
  File "/var/www/venv/lib/python3.7/site-packages/celery/bootsteps.py", line 116, in start
    step.start(parent)
  File "/var/www/venv/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 618, in start
    c.loop(*c.loop_args())
  File "/var/www/venv/lib/python3.7/site-packages/celery/worker/loops.py", line 97, in asynloop
    next(loop)
  File "/var/www/venv/lib/python3.7/site-packages/kombu/asynchronous/hub.py", line 362, in create_loop
    cb(*cbargs)
  File "/var/www/venv/lib/python3.7/site-packages/kombu/transport/redis.py", line 1266, in on_readable
    self.cycle.on_readable(fileno)
  File "/var/www/venv/lib/python3.7/site-packages/kombu/transport/redis.py", line 504, in on_readable
    chan.handlers[type]()
  File "/var/www/venv/lib/python3.7/site-packages/kombu/transport/redis.py", line 898, in _brpop_read
    **options)
  File "/var/www/venv/lib/python3.7/site-packages/redis/client.py", line 1189, in parse_response
    response = connection.read_response()
  File "/var/www/venv/lib/python3.7/site-packages/redis/connection.py", line 817, in read_response
    raise response
redis.exceptions.ReadOnlyError: You can't write against a read only replica.
```

As far as we checked, an error during connection becomes `ConnectionError` and regarded as "Recoverable", but `ReadOnlyError` (and its parent `ResponseError`) is regarded as "Unrecoverable".

# Design

## Architectural Considerations
<!--
If more components other than Celery are involved,
describe them here and the effect it would have on Celery.
-->
Maybe this part in kombu is related
https://github.com/celery/kombu/blob/v5.2.3/kombu/transport/redis.py#L128-L141

## Proposed Behavior
<!--
Please describe in detail how this enhancement is going to change the behavior
of an existing feature.
Describe what happens in case of failures as well if applicable.
-->
When an error happens due to switching master to replica, retry connection and connect to the new master.

## Proposed UI/UX
<!--
Please provide your ideas for the API, CLI options,
configuration key names etc. that will be adjusted for this enhancement.
-->
- Adding some option to regard ResponseError as "Recoverable".  Configuration key like `redis_retry_on_response_error` or `redis_retry_on_failover`?

But we're not sure if regarding ResponseError as "Recoverable" has a critical impact on existing behavior. We'd really appreciate advice from the community.

## Diagrams
<!--
Please include any diagrams that might be relevant
to the implementation of this enhancement such as:
* Class Diagrams
* Sequence Diagrams
* Activity Diagrams
You can drag and drop images into the text box to attach them to this issue.
-->
N/A

## Alternatives
<!--
If you have considered any alternative implementations
describe them in detail below.
-->
None



Discussion:
Hey @nkns165 :wave:,
Thank you for opening an issue. We will get back to you as soon as we can.
Also, check out our [Open Collective](https://opencollective.com/celery) and consider backing us - every little helps!

We also offer priority support for our sponsors.
If you require immediate assistance please consider sponsoring us.


Resolving patch:
diff --git a/celery/app/defaults.py b/celery/app/defaults.py
--- a/celery/app/defaults.py
+++ b/celery/app/defaults.py
@@ -89,6 +89,7 @@ def __repr__(self):
         connection_retry=Option(True, type='bool'),
         connection_retry_on_startup=Option(None, type='bool'),
         connection_max_retries=Option(100, type='int'),
+        channel_error_retry=Option(False, type='bool'),
         failover_strategy=Option(None, type='string'),
         heartbeat=Option(120, type='int'),
         heartbeat_checkrate=Option(3.0, type='int'),
diff --git a/celery/worker/consumer/consumer.py b/celery/worker/consumer/consumer.py
--- a/celery/worker/consumer/consumer.py
+++ b/celery/worker/consumer/consumer.py
@@ -328,9 +328,13 @@ def start(self):
                     crit('Frequent restarts detected: %r', exc, exc_info=1)
                     sleep(1)
             self.restart_count += 1
+            if self.app.conf.broker_channel_error_retry:
+                recoverable_errors = (self.connection_errors + self.channel_errors)
+            else:
+                recoverable_errors = self.connection_errors
             try:
                 blueprint.start(self)
-            except self.connection_errors as exc:
+            except recoverable_errors as exc:
                 # If we're not retrying connections, we need to properly shutdown or terminate
                 # the Celery main process instead of abruptly aborting the process without any cleanup.
                 is_connection_loss_on_startup = self.restart_count == 0


Test patch:
diff --git a/t/unit/worker/test_consumer.py b/t/unit/worker/test_consumer.py
--- a/t/unit/worker/test_consumer.py
+++ b/t/unit/worker/test_consumer.py
@@ -4,6 +4,7 @@
 from unittest.mock import MagicMock, Mock, call, patch
 
 import pytest
+from amqp import ChannelError
 from billiard.exceptions import RestartFreqExceeded
 
 from celery import bootsteps
@@ -310,6 +311,31 @@ def test_blueprint_restart_when_state_not_in_stop_conditions(self, broker_connec
         c.start()
         c.blueprint.restart.assert_called_once()
 
+    @pytest.mark.parametrize("broker_channel_error_retry", [True, False])
+    def test_blueprint_restart_for_channel_errors(self, broker_channel_error_retry):
+        c = self.get_consumer()
+
+        # ensure that WorkerShutdown is not raised
+        c.app.conf['broker_connection_retry'] = True
+        c.app.conf['broker_connection_retry_on_startup'] = True
+        c.app.conf['broker_channel_error_retry'] = broker_channel_error_retry
+        c.restart_count = -1
+
+        # ensure that blueprint state is not in stop conditions
+        c.blueprint.state = bootsteps.RUN
+        c.blueprint.start.side_effect = ChannelError()
+
+        # stops test from running indefinitely in the while loop
+        c.blueprint.restart.side_effect = self._closer(c)
+
+        # restarted only when broker_channel_error_retry is True
+        if broker_channel_error_retry:
+            c.start()
+            c.blueprint.restart.assert_called_once()
+        else:
+            with pytest.raises(ChannelError):
+                c.start()
+
     def test_collects_at_restart(self):
         c = self.get_consumer()
         c.connection.collect.side_effect = MemoryError()