Repository: celery/celery

Issue:
MongoDB result backend serialization logic issue
<!--
Please fill this template entirely and do not erase parts of it.
We reserve the right to close without a response
bug reports which are incomplete.
-->
# Checklist
<!--
To check an item on the list replace [ ] with [x].
-->
- [x] I have verified that the issue exists against the `master` branch of Celery.
- [ ] This has already been asked to the [discussion group](https://groups.google.com/forum/#!forum/celery-users) first.
- [x] I have read the relevant section in the
  [contribution guide](http://docs.celeryproject.org/en/latest/contributing.html#other-bugs)
  on reporting bugs.
- [x] I have checked the [issues list](https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22)
  for similar or identical bug reports.
- [x] I have checked the [pull requests list](https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22)
  for existing proposed fixes.
- [x] I have checked the [commit log](https://github.com/celery/celery/commits/master)
  to find out if the bug was already fixed in the master branch.
- [x] I have included all related issues and possible duplicate issues
  in this issue (If there are none, check this box anyway).

## Mandatory Debugging Information

- [ ] I have included the output of ``celery -A proj report`` in the issue.
    (if you are not able to do this, then at least specify the Celery
     version affected).
- [x] I have verified that the issue exists against the `master` branch of Celery.
- [x] I have included the contents of ``pip freeze`` in the issue.
- [x] I have included all the versions of all the external dependencies required
  to reproduce this bug.

## Optional Debugging Information
<!--
Try some of the below if you think they are relevant.
It will help us figure out the scope of the bug and how many users it affects.
-->
- [ ] I have tried reproducing the issue on more than one Python version
  and/or implementation.
- [ ] I have tried reproducing the issue on more than one message broker and/or
  result backend.
- [ ] I have tried reproducing the issue on more than one version of the message
  broker and/or result backend.
- [ ] I have tried reproducing the issue on more than one operating system.
- [ ] I have tried reproducing the issue on more than one workers pool.
- [ ] I have tried reproducing the issue with autoscaling, retries,
  ETA/Countdown & rate limits disabled.
- [ ] I have tried reproducing the issue after downgrading
  and/or upgrading Celery and its dependencies.

## Related Issues and Possible Duplicates
<!--
Please make sure to search and mention any related issues
or possible duplicates to this issue as requested by the checklist above.

This may or may not include issues in other repositories that the Celery project
maintains or other repositories that are dependencies of Celery.

If you don't know how to mention issues, please refer to Github's documentation
on the subject: https://help.github.com/en/articles/autolinked-references-and-urls#issues-and-pull-requests
-->

#### Related Issues

- In https://github.com/celery/celery/pull/5661 PR result encoding was removed from `_store_result` call.
- In https://github.com/celery/celery/pull/5918 PR missing result encoding on storing has been attempted to be resolved by adding `encode` call on `decode` call. 

#### Possible Duplicates

- None

## Environment & Settings
<!-- Include the contents of celery --version below -->
**Celery version**:
<!-- Include the output of celery -A proj report below -->
<details>
<summary><b><code>celery report</code> Output:</b></summary>
<p>

```
```

</p>
</details>

# Steps to Reproduce

## Required Dependencies
<!-- Please fill the required dependencies to reproduce this issue -->
* **Minimal Python Version**: N/A or Unknown
* **Minimal Celery Version**: N/A or Unknown
* **Minimal Kombu Version**: N/A or Unknown
* **Minimal Broker Version**: N/A or Unknown
* **Minimal Result Backend Version**: N/A or Unknown
* **Minimal OS and/or Kernel Version**: N/A or Unknown
* **Minimal Broker Client Version**: N/A or Unknown
* **Minimal Result Backend Client Version**: N/A or Unknown

### Python Packages
<!-- Please fill the contents of pip freeze below -->
<details>
<summary><b><code>pip freeze</code> Output:</b></summary>
<p>

```
```

</p>
</details>

### Other Dependencies
<!--
Please provide system dependencies, configuration files
and other dependency information if applicable
-->
<details>
<p>
N/A
</p>
</details>

## Minimally Reproducible Test Case
<!--
Please provide a reproducible test case.
Refer to the Reporting Bugs section in our contribution guide.

We prefer submitting test cases in the form of a PR to our integration test suite.
If you can provide one, please mention the PR number below.
If not, please attach the most minimal code example required to reproduce the issue below.
If the test case is too large, please include a link to a gist or a repository below.
-->

<details>
<p>

```python
```

</p>
</details>

# Expected Behavior
`store_result` uses `serializer`.

# Actual Behavior
`store_result` ignores `serializer`, so getting serialization errors for data types that are compatible with chosen serializer.

For more details please refer to tests in https://github.com/celery/celery/pull/6259



Discussion:

Resolving patch:
diff --git a/celery/backends/mongodb.py b/celery/backends/mongodb.py
--- a/celery/backends/mongodb.py
+++ b/celery/backends/mongodb.py
@@ -180,14 +180,12 @@ def encode(self, data):
     def decode(self, data):
         if self.serializer == 'bson':
             return data
-
-        payload = self.encode(data)
-        return super(MongoBackend, self).decode(payload)
+        return super(MongoBackend, self).decode(data)
 
     def _store_result(self, task_id, result, state,
                       traceback=None, request=None, **kwargs):
         """Store return value and state of an executed task."""
-        meta = self._get_result_meta(result=result, state=state,
+        meta = self._get_result_meta(result=self.encode(result), state=state,
                                      traceback=traceback, request=request)
         # Add the _id for mongodb
         meta['_id'] = task_id


Test patch:
diff --git a/t/unit/backends/test_mongodb.py b/t/unit/backends/test_mongodb.py
--- a/t/unit/backends/test_mongodb.py
+++ b/t/unit/backends/test_mongodb.py
@@ -1,9 +1,11 @@
 from __future__ import absolute_import, unicode_literals
 
 import datetime
+import sys
 from pickle import dumps, loads
 
 import pytest
+import pytz
 from case import ANY, MagicMock, Mock, mock, patch, sentinel, skip
 from kombu.exceptions import EncodeError
 try:
@@ -12,7 +14,7 @@
     ConfigurationError = None
 
 from celery import states, uuid
-from celery.backends.mongodb import InvalidDocument, MongoBackend
+from celery.backends.mongodb import InvalidDocument, MongoBackend, Binary
 from celery.exceptions import ImproperlyConfigured
 
 COLLECTION = 'taskmeta_celery'
@@ -530,20 +532,138 @@ def test_regression_worker_startup_info(self):
             assert worker.startup_info()
 
 
+@pytest.fixture(scope="function")
+def mongo_backend_factory(app):
+    """Return a factory that creates MongoBackend instance with given serializer, including BSON."""
+
+    def create_mongo_backend(serializer):
+        # NOTE: `bson` is a only mongodb-specific type and can be set only directly on MongoBackend instance.
+        if serializer == "bson":
+            beckend = MongoBackend(app=app)
+            beckend.serializer = serializer
+        else:
+            app.conf.accept_content = ['json', 'pickle', 'msgpack', 'yaml']
+            app.conf.result_serializer = serializer
+            beckend = MongoBackend(app=app)
+        return beckend
+
+    yield create_mongo_backend
+
+
 @skip.unless_module('pymongo')
+@pytest.mark.parametrize("serializer,encoded_into", [
+    ('bson', int),
+    ('json', str),
+    ('pickle', Binary),
+    ('msgpack', Binary),
+    ('yaml', str),
+])
 class test_MongoBackend_no_mock:
 
-    def test_encode_decode(self, app):
-        backend = MongoBackend(app=app)
-        data = {'foo': 1}
-        assert backend.decode(backend.encode(data))
-        backend.serializer = 'bson'
-        assert backend.encode(data) == data
-        assert backend.decode(data) == data
-
-    def test_de(self, app):
-        backend = MongoBackend(app=app)
-        data = {'foo': 1}
-        assert backend.encode(data)
-        backend.serializer = 'bson'
-        assert backend.encode(data) == data
+    def test_encode(self, mongo_backend_factory, serializer, encoded_into):
+        backend = mongo_backend_factory(serializer=serializer)
+        assert isinstance(backend.encode(10), encoded_into)
+
+    def test_encode_decode(self, mongo_backend_factory, serializer, encoded_into):
+        backend = mongo_backend_factory(serializer=serializer)
+        decoded = backend.decode(backend.encode(12))
+        assert decoded == 12
+
+
+class _MyTestClass(object):
+
+    def __init__(self, a):
+        self.a = a
+
+    def __eq__(self, other):
+        assert self.__class__ == type(other)
+        return self.a == other.a
+
+
+SUCCESS_RESULT_TEST_DATA = [
+    # json types
+    {
+        "result": "A simple string",
+        "serializers": ["bson", "pickle", "yaml", "json", "msgpack"],
+    },
+    {
+        "result": 100,
+        "serializers": ["bson", "pickle", "yaml", "json", "msgpack"],
+    },
+    {
+        "result": 9.1999999999999999,
+        "serializers": ["bson", "pickle", "yaml", "json", "msgpack"],
+    },
+    {
+        "result": {"foo": "simple result"},
+        "serializers": ["bson", "pickle", "yaml", "json", "msgpack"],
+    },
+    {
+        "result": ["a", "b"],
+        "serializers": ["bson", "pickle", "yaml", "json", "msgpack"],
+    },
+    {
+        "result": False,
+        "serializers": ["bson", "pickle", "yaml", "json", "msgpack"],
+    },
+    {
+        "result": None,
+        "serializers": ["bson", "pickle", "yaml", "json", "msgpack"],
+    },
+    # advanced essential types
+    {
+        "result": datetime.datetime(2000, 1, 1, 0, 0, 0, 0),
+        "serializers": ["bson", "pickle", "yaml"],
+    },
+    {
+        "result": datetime.datetime(2000, 1, 1, 0, 0, 0, 0, tzinfo=pytz.utc),
+        "serializers": ["pickle", "yaml"],
+    },
+    # custom types
+    {
+        "result": _MyTestClass("Hi!"),
+        "serializers": ["pickle"],
+    },
+]
+
+
+@skip.unless_module('pymongo')
+class test_MongoBackend_store_get_result:
+
+    @pytest.fixture(scope="function", autouse=True)
+    def fake_mongo_collection_patch(self, monkeypatch):
+        """A fake collection with serialization experience close to MongoDB."""
+        bson = pytest.importorskip("bson")
+
+        class FakeMongoCollection(object):
+            def __init__(self):
+                self.data = {}
+
+            def replace_one(self, task_id, meta, upsert=True):
+                self.data[task_id['_id']] = bson.encode(meta)
+
+            def find_one(self, task_id):
+                return bson.decode(self.data[task_id['_id']])
+
+        monkeypatch.setattr(MongoBackend, "collection", FakeMongoCollection())
+
+    @pytest.mark.parametrize("serializer,result_type,result", [
+        (s, type(i['result']), i['result']) for i in SUCCESS_RESULT_TEST_DATA for s in i['serializers']]
+    )
+    def test_encode_success_results(self, mongo_backend_factory, serializer, result_type, result):
+        backend = mongo_backend_factory(serializer=serializer)
+        backend.store_result(TASK_ID, result, 'SUCCESS')
+        recovered = backend.get_result(TASK_ID)
+        if sys.version_info.major == 2 and isinstance(recovered, str):
+            result_type = str  # workaround for python 2 compatibility and `unicode_literals`
+        assert type(recovered) == result_type
+        assert recovered == result
+
+    @pytest.mark.parametrize("serializer", ["bson", "pickle", "yaml", "json", "msgpack"])
+    def test_encode_exception_error_results(self, mongo_backend_factory, serializer):
+        backend = mongo_backend_factory(serializer=serializer)
+        exception = Exception("Basic Exception")
+        backend.store_result(TASK_ID, exception, 'FAILURE')
+        recovered = backend.get_result(TASK_ID)
+        assert type(recovered) == type(exception)
+        assert recovered.args == exception.args