Repository: celery/celery

Issue:
`publish_result` is False when tasks are eager and and we DON'T want to ignore results
 #### Mandatory Debugging Information

- [ ] I have included the output of ``celery -A proj report`` in the issue.
    (if you are not able to do this, then at least specify the Celery
     version affected).
- [x] I have verified that the issue exists against the `master` branch of Celery.
- [ ] I have included the contents of ``pip freeze`` in the issue.
- [x] I have included all the versions of all the external dependencies required
  to reproduce this bug.

#### Related Issues

- #5398 
- https://github.com/celery/django-celery-results/issues/49

#### Possible Duplicates

- None

### Python Packages

Latest, not released django-celery-results
https://github.com/celery/django-celery-results/pull/172/commits/b4acbf24ea3521ecfdbf821ece7ca62e49ff2765
Currently published version doesn't support Celery 5.x, but all versions are affected


## Minimally Reproducible Test Case

Source of the issue
https://github.com/celery/celery/blob/2a6c7cfe3b1283961887bf1cb3f5aa6c8aa70820/celery/app/trace.py#L323

```python
import pytest

@pytest.mark.parametrize(
    "eager,ignore_result,publish_result",
    [
        (True, True, False),
        (True, False, True),  # fails
        (False, True, False),
        (False, False, True),
    ]

)
def test_publish_result_current(eager, ignore_result, publish_result):
    actual_publish_result = not eager and not ignore_result
    assert publish_result == actual_publish_result


@pytest.mark.parametrize(
    "eager,ignore_result,publish_result",
    [
        (True, True, False),
        (True, False, True),
        (False, True, False),
        (False, False, True),
    ]

)
def test_publish_result_desired(eager, ignore_result, publish_result):
    actual_publish_result = not ignore_result
    assert publish_result == actual_publish_result
```

# Expected Behavior

``always_eager`` exists so we can use Celery in unit tests. I want to execute a task eagerly and later on check result from the backend. 

# Actual Behavior
When task is eager and it's set NOT to ignore the result, the result will be ignored.


Discussion:
did you try the latest release & share your feedback?
it would be great if you can share a possible solution
I tried the latest release.

It appears to me that someone made the decision that whenever a task is eager, nothing should be saved to the backend. But not only I struggle to understand what would be the purpose of this feature but also it's not documented anywhere (AFAIK). So could you please confirm that the current behavior is not the desired behavior?

Once you confirm I'll be happy to fix it.
I think we do need to save the result. However, if this was never done before in any version of Celery then this is not a bug but a feature request.

I'd like to have some input from @celery/core-developers.
> I tried the latest release.
> 
> It appears to me that someone made the decision that whenever a task is eager, nothing should be saved to the backend. But not only I struggle to understand what would be the purpose of this feature but also it's not documented anywhere (AFAIK). So could you please confirm that the current behavior is not the desired behavior?
> 
> Once you confirm I'll be happy to fix it.

based on the issues raised https://github.com/celery/django-celery-results/issues/49 I believe we should save the results, however, we should carefully check the possible side effects.
@tomwojcik still up for this?
Yes @auvipy . I already tried to fix this but some tests started to fail due to some side effects but I think some are to be expected. I will give it another try over the weekend and let you know if I won't manage to fix this myself.
I personally, haven't used eager execution of the task. But if I understand it correctly that it execute task locally without sending it to the queue, it makes sense to me that the result is not stored to result storage (since it is executed directly by the caller). But still haven't used it so maybe there is need for storing it in result store.
> Yes @auvipy . I already tried to fix this but some tests started to fail due to some side effects but I think some are to be expected. I will give it another try over the weekend and let you know if I won't manage to fix this myself.

don't hesitate to come with A draft PR as well. so that if you are not sure about something, we could share our insight with you.

Resolving patch:
diff --git a/celery/app/defaults.py b/celery/app/defaults.py
--- a/celery/app/defaults.py
+++ b/celery/app/defaults.py
@@ -255,6 +255,7 @@ def __repr__(self):
             False, type='bool', old={'celery_eager_propagates_exceptions'},
         ),
         ignore_result=Option(False, type='bool'),
+        store_eager_result=Option(False, type='bool'),
         protocol=Option(2, type='int', old={'celery_task_protocol'}),
         publish_retry=Option(
             True, type='bool', old={'celery_task_publish_retry'},
diff --git a/celery/app/task.py b/celery/app/task.py
--- a/celery/app/task.py
+++ b/celery/app/task.py
@@ -309,6 +309,7 @@ class Task:
         ('acks_on_failure_or_timeout', 'task_acks_on_failure_or_timeout'),
         ('reject_on_worker_lost', 'task_reject_on_worker_lost'),
         ('ignore_result', 'task_ignore_result'),
+        ('store_eager_result', 'task_store_eager_result'),
         ('store_errors_even_if_ignored', 'task_store_errors_even_if_ignored'),
     )
 
diff --git a/celery/app/trace.py b/celery/app/trace.py
--- a/celery/app/trace.py
+++ b/celery/app/trace.py
@@ -159,9 +159,13 @@ def __init__(self, state, retval=None):
 
     def handle_error_state(self, task, req,
                            eager=False, call_errbacks=True):
-        store_errors = not eager
         if task.ignore_result:
             store_errors = task.store_errors_even_if_ignored
+        elif eager and task.store_eager_result:
+            store_errors = True
+        else:
+            store_errors = not eager
+
         return {
             RETRY: self.handle_retry,
             FAILURE: self.handle_failure,
@@ -316,7 +320,13 @@ def build_tracer(name, task, loader=None, hostname=None, store_errors=True,
     ignore_result = task.ignore_result
     track_started = task.track_started
     track_started = not eager and (task.track_started and not ignore_result)
-    publish_result = not eager and not ignore_result
+
+    # #6476
+    if eager and not ignore_result and task.store_eager_result:
+        publish_result = True
+    else:
+        publish_result = not eager and not ignore_result
+
     hostname = hostname or gethostname()
     inherit_parent_priority = app.conf.task_inherit_parent_priority
 


Test patch:
diff --git a/docs/userguide/testing.rst b/docs/userguide/testing.rst
--- a/docs/userguide/testing.rst
+++ b/docs/userguide/testing.rst
@@ -18,6 +18,9 @@ To test task behavior in unit tests the preferred method is mocking.
     of what happens in a worker, and there are many discrepancies
     between the emulation and what happens in reality.
 
+    Note that eagerly executed tasks don't write results to backend by default.
+    If you want to enable this functionality, have a look at :setting:`task_store_eager_result`.
+
 A Celery task is much like a web view, in that it should only
 define how to perform the action in the context of being called as a task.
 
diff --git a/t/unit/tasks/test_trace.py b/t/unit/tasks/test_trace.py
--- a/t/unit/tasks/test_trace.py
+++ b/t/unit/tasks/test_trace.py
@@ -1,4 +1,4 @@
-from unittest.mock import Mock, patch
+from unittest.mock import ANY, Mock, patch
 
 import pytest
 from billiard.einfo import ExceptionInfo
@@ -148,6 +148,75 @@ def add(x, y):
         with pytest.raises(MemoryError):
             self.trace(add, (2, 2), {}, eager=False)
 
+    def test_eager_task_does_not_store_result_even_if_not_ignore_result(self):
+        @self.app.task(shared=False)
+        def add(x, y):
+            return x + y
+
+        add.backend = Mock(name='backend')
+        add.ignore_result = False
+
+        self.trace(add, (2, 2), {}, eager=True)
+
+        add.backend.mark_as_done.assert_called_once_with(
+            'id-1',     # task_id
+            4,          # result
+            ANY,        # request
+            False       # store_result
+        )
+
+    def test_eager_task_does_not_call_store_result(self):
+        @self.app.task(shared=False)
+        def add(x, y):
+            return x + y
+
+        backend = BaseDictBackend(app=self.app)
+        backend.store_result = Mock()
+        add.backend = backend
+        add.ignore_result = False
+
+        self.trace(add, (2, 2), {}, eager=True)
+
+        add.backend.store_result.assert_not_called()
+
+    def test_eager_task_will_store_result_if_proper_setting_is_set(self):
+        @self.app.task(shared=False)
+        def add(x, y):
+            return x + y
+
+        add.backend = Mock(name='backend')
+        add.store_eager_result = True
+        add.ignore_result = False
+
+        self.trace(add, (2, 2), {}, eager=True)
+
+        add.backend.mark_as_done.assert_called_once_with(
+            'id-1',     # task_id
+            4,          # result
+            ANY,        # request
+            True        # store_result
+        )
+
+    def test_eager_task_with_setting_will_call_store_result(self):
+        @self.app.task(shared=False)
+        def add(x, y):
+            return x + y
+
+        backend = BaseDictBackend(app=self.app)
+        backend.store_result = Mock()
+        add.backend = backend
+        add.store_eager_result = True
+        add.ignore_result = False
+
+        self.trace(add, (2, 2), {}, eager=True)
+
+        add.backend.store_result.assert_called_once_with(
+            'id-1',
+            4,
+            states.SUCCESS,
+            request=ANY
+        )
+
     def test_when_backend_raises_exception(self):
         @self.app.task(shared=False)
         def add(x, y):
@@ -413,6 +482,32 @@ def test_handle_error_state(self):
             call_errbacks=True,
         )
 
+    def test_handle_error_state_for_eager_task(self):
+        x = self.TI(states.FAILURE)
+        x.handle_failure = Mock()
+
+        x.handle_error_state(self.add, self.add.request, eager=True)
+        x.handle_failure.assert_called_once_with(
+            self.add,
+            self.add.request,
+            store_errors=False,
+            call_errbacks=True,
+        )
+
+    def test_handle_error_for_eager_saved_to_backend(self):
+        x = self.TI(states.FAILURE)
+        x.handle_failure = Mock()
+
+        self.add.store_eager_result = True
+
+        x.handle_error_state(self.add, self.add.request, eager=True)
+        x.handle_failure.assert_called_with(
+            self.add,
+            self.add.request,
+            store_errors=True,
+            call_errbacks=True,
+        )
+
     @patch('celery.app.trace.ExceptionInfo')
     def test_handle_reject(self, ExceptionInfo):
         x = self.TI(states.FAILURE)