Repository: celery/celery

Issue:
Workers retry Redis connection when task result is larger than 512MB - retry is useless as it never fits to Redis
<!--
Please fill this template entirely and do not erase parts of it.
We reserve the right to close without a response
bug reports which are incomplete.
-->
# Checklist
<!--
To check an item on the list replace [ ] with [x].
-->
- [x] I have verified that the issue exists against the `master` branch of Celery.
- [ ] This has already been asked to the [discussion group](https://groups.google.com/forum/#!forum/celery-users) first.
- [x] I have read the relevant section in the
  [contribution guide](http://docs.celeryproject.org/en/latest/contributing.html#other-bugs)
  on reporting bugs.
- [x] I have checked the [issues list](https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22)
  for similar or identical bug reports.
- [x] I have checked the [pull requests list](https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22)
  for existing proposed fixes.
- [x] I have checked the [commit log](https://github.com/celery/celery/commits/master)
  to find out if the bug was already fixed in the master branch.
- [x] I have included all related issues and possible duplicate issues
  in this issue (If there are none, check this box anyway).

## Mandatory Debugging Information

- [x] I have included the output of ``celery -A proj report`` in the issue.
    (if you are not able to do this, then at least specify the Celery
     version affected).
- [x] I have verified that the issue exists against the `master` branch of Celery.
- [x] I have included the contents of ``pip freeze`` in the issue.
- [x] I have included all the versions of all the external dependencies required
  to reproduce this bug.

## Optional Debugging Information
<!--
Try some of the below if you think they are relevant.
It will help us figure out the scope of the bug and how many users it affects.
-->
- [ ] I have tried reproducing the issue on more than one Python version
  and/or implementation.
- [ ] I have tried reproducing the issue on more than one message broker and/or
  result backend.
- [ ] I have tried reproducing the issue on more than one version of the message
  broker and/or result backend.
- [ ] I have tried reproducing the issue on more than one operating system.
- [ ] I have tried reproducing the issue on more than one workers pool.
- [ ] I have tried reproducing the issue with autoscaling, retries,
  ETA/Countdown & rate limits disabled.
- [ ] I have tried reproducing the issue after downgrading
  and/or upgrading Celery and its dependencies.

## Related Issues and Possible Duplicates
<!--
Please make sure to search and mention any related issues
or possible duplicates to this issue as requested by the checklist above.

This may or may not include issues in other repositories that the Celery project
maintains or other repositories that are dependencies of Celery.

If you don't know how to mention issues, please refer to Github's documentation
on the subject: https://help.github.com/en/articles/autolinked-references-and-urls#issues-and-pull-requests
-->

#### Related Issues

- None

#### Possible Duplicates

- None

## Environment & Settings
<!-- Include the contents of celery --version below -->
**Celery version**: 5.0.4 (singularity)
<!-- Include the output of celery -A proj report below -->
<details>
<summary><b><code>celery report</code> Output:</b></summary>
<p>

```
software -> celery:5.0.4 (singularity) kombu:5.0.2 py:3.9.0
            billiard:3.6.3.0 py-amqp:5.0.2
platform -> system:Linux arch:64bit, ELF
            kernel version:5.9.12-arch1-1 imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:amqp results:redis://:**@****:6379/10

broker_url: 'amqp://**:********@*****:5672/**'
result_backend: 'redis://:********@*****:6379/10'
deprecated_settings: None
```

</p>
</details>

# Steps to Reproduce

## Required Dependencies
<!-- Please fill the required dependencies to reproduce this issue -->
* **Minimal Python Version**: N/A or Unknown
* **Minimal Celery Version**: N/A or Unknown
* **Minimal Kombu Version**: N/A or Unknown
* **Minimal Broker Version**: N/A or Unknown
* **Minimal Result Backend Version**: N/A or Unknown
* **Minimal OS and/or Kernel Version**: N/A or Unknown
* **Minimal Broker Client Version**: N/A or Unknown
* **Minimal Result Backend Client Version**: N/A or Unknown

### Python Packages
<!-- Please fill the contents of pip freeze below -->
<details>
<summary><b><code>pip freeze</code> Output:</b></summary>
<p>

```
amqp==5.0.2
billiard==3.6.3.0
celery @ git+https://github.com/celery/celery.git@420e3931a63538bd225ef57916deccf53cbcb57a  // == master, tried also 5.0.4
click==7.1.2
click-didyoumean==0.0.3
click-plugins==1.1.1
click-repl==0.1.6
kombu==5.0.2
prompt-toolkit==3.0.8
pytz==2020.4
redis==3.5.3
six==1.15.0
vine==5.0.0
wcwidth==0.2.5
```

</p>
</details>

### Other Dependencies
<!--
Please provide system dependencies, configuration files
and other dependency information if applicable
-->
<details>
<p>
N/A
</p>
</details>

## Minimally Reproducible Test Case
<!--
Please provide a reproducible test case.
Refer to the Reporting Bugs section in our contribution guide.

We prefer submitting test cases in the form of a PR to our integration test suite.
If you can provide one, please mention the PR number below.
If not, please attach the most minimal code example required to reproduce the issue below.
If the test case is too large, please include a link to a gist or a repository below.
-->

<details>
<p>

```python
#!/usr/bin/env python3

from celery import Celery

app = Celery(
    'tasks',
    broker='amqp://user:***@**:5672/**',
    backend='redis://:**@**:6379/1',
)

@app.task(ignore_result=False)
def test(*args, **kwargs):
    return 'x' * 536870911  # 512MB
```

</p>
</details>

# Expected Behavior
<!-- Describe in detail what you expect to happen -->

I'd except an exception or error. There is no point in retrying storing result to Redis when it simply never fits there. String limit is 512 MB.

I could check size of the data I'm returning in task. However, Celery adds additional metadata to my result so I do not know to what size should I limit my result.

# Actual Behavior
<!--
Describe in detail what actually happened.
Please include a backtrace and surround it with triple backticks (```).
In addition, include the Celery daemon logs, the broker logs,
the result backend logs and system logs below if they will help us debug
the issue.
-->
```
[2020-12-09 08:38:52,786: ERROR/ForkPoolWorker-8] Connection to Redis lost: Retry (0/20) now.
[2020-12-09 08:38:53,328: ERROR/ForkPoolWorker-8] Connection to Redis lost: Retry (1/20) in 1.00 second.
[2020-12-09 08:38:54,940: ERROR/ForkPoolWorker-8] Connection to Redis lost: Retry (2/20) in 1.00 second.
[2020-12-09 08:38:56,547: ERROR/ForkPoolWorker-8] Connection to Redis lost: Retry (3/20) in 1.00 second.
.... and so on
```

Redis fails with the following error when you try to set string larger than 512 MB:
```
raise ConnectionError("Error %s while writing to socket. %s." %
redis.exceptions.ConnectionError: Error 32 while writing to socket. Broken pipe.
```
Tried via another script with direct Redis connection. This says nothing about input size therefore I think value size should be checked before sending data to Redis.



Discussion:
Pull requests are welcome.

Resolving patch:
diff --git a/celery/backends/base.py b/celery/backends/base.py
--- a/celery/backends/base.py
+++ b/celery/backends/base.py
@@ -853,7 +853,11 @@ def _store_result(self, task_id, result, state,
         if current_meta['status'] == states.SUCCESS:
             return result
 
-        self._set_with_state(self.get_key_for_task(task_id), self.encode(meta), state)
+        try:
+            self._set_with_state(self.get_key_for_task(task_id), self.encode(meta), state)
+        except BackendStoreError as ex:
+            raise BackendStoreError(str(ex), state=state, task_id=task_id) from ex
+
         return result
 
     def _save_group(self, group_id, result):
diff --git a/celery/backends/redis.py b/celery/backends/redis.py
--- a/celery/backends/redis.py
+++ b/celery/backends/redis.py
@@ -12,7 +12,7 @@
 from celery import states
 from celery._state import task_join_will_block
 from celery.canvas import maybe_signature
-from celery.exceptions import ChordError, ImproperlyConfigured
+from celery.exceptions import BackendStoreError, ChordError, ImproperlyConfigured
 from celery.result import GroupResult, allow_join_result
 from celery.utils.functional import dictfilter
 from celery.utils.log import get_logger
@@ -192,6 +192,10 @@ class RedisBackend(BaseKeyValueStoreBackend, AsyncBackendMixin):
     supports_autoexpire = True
     supports_native_join = True
 
+    #: Maximal length of string value in Redis.
+    #: 512 MB - https://redis.io/topics/data-types
+    _MAX_STR_VALUE_SIZE = 536870912
+
     def __init__(self, host=None, port=None, db=None, password=None,
                  max_connections=None, url=None,
                  connection_pool=None, **kwargs):
@@ -364,6 +368,9 @@ def on_connection_error(self, max_retries, exc, intervals, retries):
         return tts
 
     def set(self, key, value, **retry_policy):
+        if len(value) > self._MAX_STR_VALUE_SIZE:
+            raise BackendStoreError('value too large for Redis backend')
+
         return self.ensure(self._set, (key, value), **retry_policy)
 
     def _set(self, key, value):
diff --git a/celery/exceptions.py b/celery/exceptions.py
--- a/celery/exceptions.py
+++ b/celery/exceptions.py
@@ -288,7 +288,7 @@ def __repr__(self):
 
 
 class BackendStoreError(BackendError):
-    """An issue writing from the backend."""
+    """An issue writing to the backend."""
 
     def __init__(self, *args, **kwargs):
         self.state = kwargs.get('state', "")


Test patch:
diff --git a/t/unit/backends/test_redis.py b/t/unit/backends/test_redis.py
--- a/t/unit/backends/test_redis.py
+++ b/t/unit/backends/test_redis.py
@@ -12,7 +12,7 @@
 
 from celery import signature, states, uuid
 from celery.canvas import Signature
-from celery.exceptions import ChordError, ImproperlyConfigured
+from celery.exceptions import BackendStoreError, ChordError, ImproperlyConfigured
 from celery.utils.collections import AttributeDict
 
 
@@ -675,6 +675,10 @@ def test_set_expires(self):
             key, 512,
         )
 
+    def test_set_raises_error_on_large_value(self):
+        with pytest.raises(BackendStoreError):
+            self.b.set('key', 'x' * (self.b._MAX_STR_VALUE_SIZE + 1))
+
 
 class test_RedisBackend_chords_simple(basetest_RedisBackend):
     @pytest.fixture(scope="class", autouse=True)