Repository: celery/celery

Issue:
Eager mode hides serialization side-effects
Running a task in eager mode doesn't pass the arguments through serialization, and therefore hides some of its side-effects. For example, calling a celery task in eager mode with UUID arguments, will leave them as UUIDs, whereas passing them through serialization (with eager mode switched off) converts them to strings. This hid a bunch of bugs in our application code.

Do you agree it would be more reasonable to pass the arguments through the serialization backends, even in eager mode, to have a more consistently behaving API?


Discussion:
Sounds totally crazy but it's true. Passing a native date that I usually need to parse from a string, the date remains native in eager mode, breaking the functionality !
Just got it by that as well with a nested data structure containing a non JSON serializable object.
Any update on this?
FWIW we worked around the issue by making `app.Task` a subclass that performs a round trip when `CELERY_ALWAYS_EAGER is True`.

```python
from kombu.serialization import registry

app = Celery('app')

class EagerSerializationTask(app.Task):
    abstract = True

    @classmethod
    def apply_async(cls, args=None, kwargs=None, *args_, **kwargs_):
        app = cls._get_app()
        if app.conf.CELERY_ALWAYS_EAGER and kwargs_.pop('serialize', True):
            # Perform a noop serialization backtrip to assert args and kwargs
            # will be serialized appropriately when an async call through kombu
            # is actually performed. This is done to make sure we catch the
            # serializations errors with our test suite which runs with the
            # CELERY_ALWAYS_EAGER setting set to True. See the following Celery
            # issue for details https://github.com/celery/celery/issues/4008.
            producer = kwargs.get('producer') if kwargs else None
            with app.producer_or_acquire(producer) as producer:
                serializer = kwargs_.get('serializer', producer.serializer)
                args_content_type, args_content_encoding, args_data = registry.encode(args, serializer)
                kwargs_content_type, kwargs_content_encoding, kwargs_data = registry.encode(kwargs, serializer)
                args = registry.decode(args_data, args_content_type, args_content_encoding)
                kwargs = registry.decode(kwargs_data, kwargs_content_type, kwargs_content_encoding)
        return super(EagerSerializationTask, cls).apply_async(args=args, kwargs=kwargs, *args_, **kwargs_)

app.Task = EagerSerializationTask
```
Just had an issue with it today : / Big problem.
I was just debugging this issue myself with passing a datetime.date object and am now wondering how many other places I have passing tests which don't test what is actually happening in real world use.
I ran into this issue as well (with celery 4.x). Working on a monkey patch for `apply_async` for tests until this can get handled.
Does anyone of the commenters here have the knowledge, time and motivation to open a PR to remediate this issue?
@Korijn I'd happily submit a PR with the above solution if it's deemed appropriate by the maintainers of the project.
Hi Simon, I've only been a maintainer for about a week, but for what it's worth I support this 100% for calls to `delay` and `apply_async` with ALWAYS_EAGER enabled.
I'll see if I can swing some time at work to take a crack at it next week.  Not a ton of free time outside of work this time of year, unfortunately.
I was digging a bit myself to understand it all better and, not surprisingly, the conclusion I came to was that I pretty much just needed to drop @charettes' code into `apply_async()` before it calls `self.apply()`.  I do wonder if `apply()` should be doing it instead so that when called directly the data it passes to tasks will be consistent with `apply_async()`.
@jmichalicek I think the change should only be done for `delay()`/`apply_async()` because AFAIK `task.apply()` is an equivalent of `task()` and doesn't perform a serialization roundtrip for the sake of preserving provided `args`/`kwargs` identity.
I agree with leaving the existing behaviour of apply() alone. A patch should only address the case of using delay() and apply_async() while always_eager is enabled.

On Sun, 17 Dec 2017 at 00:28 Simon Charette <notifications@github.com>
wrote:

> @jmichalicek <https://github.com/jmichalicek> I think the change should
> only be done for delay()/apply_async() because AFAIK task.apply() is an
> equivalent of task() and doesn't perform a serialization roundtrip for
> the sake of preserving provided args/kwargs identity.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/celery/celery/issues/4008#issuecomment-352193745>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AA9vG5tFEfZhGk9_1C7qpk49-kufu1Z7ks5tA--tgaJpZM4NOC7e>
> .
>

@charettes @AlexHill Makes sense if `apply()` is intended to be conceptually different from `apply_async()` beyond just that one is async and the other is not.

I have always thought of them as "do the exact same thing, except one of these runs async in a separate process".  They are calling the same task with the same inputs so to me should pass the same data into the task.  But if I misunderstood the use case, then leaving `apply()` itself alone totally makes sense.  I pretty much never use `apply()`, anyway.  If I wanted it to run inline, I'd just call the function directly - I pretty much always just use the decorator on a function and that is usually just a light wrapper around calling a different function or method.

Resolving patch:
diff --git a/celery/app/task.py b/celery/app/task.py
--- a/celery/app/task.py
+++ b/celery/app/task.py
@@ -5,6 +5,7 @@
 import sys
 
 from billiard.einfo import ExceptionInfo
+from kombu import serialization
 from kombu.exceptions import OperationalError
 from kombu.utils.uuid import uuid
 
@@ -514,6 +515,17 @@ def apply_async(self, args=None, kwargs=None, task_id=None, producer=None,
 
         app = self._get_app()
         if app.conf.task_always_eager:
+            with app.producer_or_acquire(producer) as eager_producer:
+                serializer = options.get(
+                    'serializer', eager_producer.serializer
+                )
+                body = args, kwargs
+                content_type, content_encoding, data = serialization.dumps(
+                    body, serializer
+                )
+                args, kwargs = serialization.loads(
+                    data, content_type, content_encoding
+                )
             with denied_join_result():
                 return self.apply(args, kwargs, task_id=task_id or uuid(),
                                   link=link, link_error=link_error, **options)


Test patch:
diff --git a/t/unit/app/test_builtins.py b/t/unit/app/test_builtins.py
--- a/t/unit/app/test_builtins.py
+++ b/t/unit/app/test_builtins.py
@@ -94,7 +94,9 @@ def setup(self):
         self.maybe_signature = self.patching('celery.canvas.maybe_signature')
         self.maybe_signature.side_effect = pass1
         self.app.producer_or_acquire = Mock()
-        self.app.producer_or_acquire.attach_mock(ContextMock(), 'return_value')
+        self.app.producer_or_acquire.attach_mock(
+            ContextMock(serializer='json'), 'return_value'
+        )
         self.app.conf.task_always_eager = True
         self.task = builtins.add_group_task(self.app)
         BuiltinsCase.setup(self)
diff --git a/t/unit/tasks/test_tasks.py b/t/unit/tasks/test_tasks.py
--- a/t/unit/tasks/test_tasks.py
+++ b/t/unit/tasks/test_tasks.py
@@ -7,6 +7,7 @@
 import pytest
 from case import ANY, ContextMock, MagicMock, Mock, patch
 from kombu import Queue
+from kombu.exceptions import EncodeError
 
 from celery import Task, group, uuid
 from celery.app.task import _reprtask
@@ -824,6 +825,13 @@ def common_send_task_arguments(self):
             ignore_result=False
         )
 
+    def test_eager_serialization_failure(self):
+        @self.app.task
+        def task(*args, **kwargs):
+            pass
+        with pytest.raises(EncodeError):
+            task.apply_async((1, 2, 3, 4, {1}))
+
     def test_task_with_ignored_result(self):
         with patch.object(self.app, 'send_task') as send_task:
             self.task_with_ignored_result.apply_async()