Issue 1:
Why are eager task results not published to the task backend?

celery/celery/app/trace.py

Line 275
in
816ff03

publish_result = not eager and not ignore_result

I don't think this is a good design choice. It disrupts the code flow, requiring to add more logic to handle eager results. For example, my code will do something like this:

async_result = my_task.delay()
handle_task_result(async_result.id)

def handle_task_result(id):
# do stuff with the result, either synchronously or asynchronously
# the result can only be accessed through `AsyncResult(id)`

This won't work at all when eager is turned on; trying AsyncResult(id) within handle_task_result will return non existing 'PENDING' garbage. Instead, I have to add a workaround in the calling code

async_result = my_task.delay()
if async_result.is_ready():
# this is entered only when celery eager is on
# forces to deal with the result synchronously
else:
handle_task_result(async_result.id)

This additional body of code is 1. totally useless under regular circumstances, it only has value when debugging with eager on and 2. ugly, since the two ways to get results use a totally different API. If eager task results were published instead, the application code would not need to be modified at all.

Thoughts?

Issue 2:
Short description

Getting the result of a task according to the FAQ instructions fails in a non-obvious way when CELERY_ALWAYS_EAGER is True.

Full description

In celery.app.task.Task.apply_async, if CELERY_ALWAYS_EAGER is True, Task.apply is called instead of app.base.Celery.send_task. send_task hits the backend store (the queue) but apply just runs the task locally.

This is fine (but could be improved upon - see "Solution Option 3", below). It makes sense that retrieving the status of a task from the backend would not be possible if the backend is never touched.

Unfortunately, functions for checking state fail in a non-obvious way that can be confusing to users.

The base function (see celery.backends.database.DatabaseBackend._get_task_meta_for) defaults to returning a new Task object with status = states.PENDING and result = None if nothing is found in the backend. This function is called by BaseBackend.get_task_meta, which is called by BaseBackend.get_status, which is the function that is used in the AsyncResult.state property to get task state, and also in AsyncResult.ready().

Thus, for tasks which the system does not know about, the result of a query for their state is not an Exception, but a new object with a valid state and empty results field.

Possible Solutions

Solution Option 1

I think the most straightforward solution is to be explicit and raise an error if the user attempt to check the state of a task via the backend when CELERY_ALWAYS_EAGER is True.

This could be fixed by raising an exception (possibly QueueNotFound, maybe a new exception) in celery.backends.base.BaseBackend.get_task_meta if CELERY_ALWAYS_EAGER is True. For example:

if self.conf.CELERY_ALWAYS_EAGER:
raise QueueNotFound("Cannot access backend when CELERY_ALWAYS_EAGER is True")

Without this change, this FAQ response can be a bit confusing:

How do I get the result of a task if I have the ID that points there?

Answer: Use Task.AsyncResult:

When CELERY_ALWAYS_EAGER is True, it looks like the task is hanging because the state is always PENDING, even though the execution may have succeeded.

Solution Option 2

Another path forward is to update the docs to explicitly mention that a task cannot be retrieved from celery when running with CELERY_ALWAYS_EAGER. The user must store the result of running apply_async on the task.

I don't consider this a real solution, but more of a stopgap if more discussion is required.

Solution Option 3

Store EagerResult objects in celery's cache and allow the user to retrieve results from there. Since celery is already looking in the cache in BaseBackend.get_task_meta this should be reasonable to implement.

Conclusion

I hope this format of issue is acceptable. I didn't see where this specific scenario fit in the Contribution Guidelines.

Please let me know if more documentation is required here, or if an example application demonstrating the issue would be helpful.