Repository: celery/celery

Issue:
New Django task with transaction atomic, return `None` instead of the task UUID
<!--
Please fill this template entirely and do not erase parts of it.
We reserve the right to close without a response
bug reports which are incomplete.
-->
# Checklist
<!--
To check an item on the list replace [ ] with [x].
-->
- [x] I have verified that the issue exists against the `main` branch of Celery.
- [ ] This has already been asked to the [discussions forum](https://github.com/celery/celery/discussions) first.
- [x] I have read the relevant section in the
  [contribution guide](https://docs.celeryq.dev/en/main/contributing.html#other-bugs)
  on reporting bugs.
- [x] I have checked the [issues list](https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22)
  for similar or identical bug reports.
- [x] I have checked the [pull requests list](https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22)
  for existing proposed fixes.
- [x] I have checked the [commit log](https://github.com/celery/celery/commits/main)
  to find out if the bug was already fixed in the main branch.
- [ ] I have included all related issues and possible duplicate issues
  in this issue (If there are none, check this box anyway).
- [ ] I have tried to reproduce the issue with [pytest-celery](https://docs.celeryq.dev/projects/pytest-celery/en/latest/userguide/celery-bug-report.html) and added the reproduction script below.

## Mandatory Debugging Information

- [ ] I have included the output of ``celery -A proj report`` in the issue.
    (if you are not able to do this, then at least specify the Celery
     version affected).
- [x] I have verified that the issue exists against the `main` branch of Celery.
- [ ] I have included the contents of ``pip freeze`` in the issue.
- [ ] I have included all the versions of all the external dependencies required
  to reproduce this bug.

## Optional Debugging Information
<!--
Try some of the below if you think they are relevant.
It will help us figure out the scope of the bug and how many users it affects.
-->
- [ ] I have tried reproducing the issue on more than one Python version
  and/or implementation.
- [ ] I have tried reproducing the issue on more than one message broker and/or
  result backend.
- [ ] I have tried reproducing the issue on more than one version of the message
  broker and/or result backend.
- [ ] I have tried reproducing the issue on more than one operating system.
- [ ] I have tried reproducing the issue on more than one workers pool.
- [ ] I have tried reproducing the issue with autoscaling, retries,
  ETA/Countdown & rate limits disabled.
- [ ] I have tried reproducing the issue after downgrading
  and/or upgrading Celery and its dependencies.

## Related Issues and Possible Duplicates
<!--
Please make sure to search and mention any related issues
or possible duplicates to this issue as requested by the checklist above.

This may or may not include issues in other repositories that the Celery project
maintains or other repositories that are dependencies of Celery.

If you don't know how to mention issues, please refer to Github's documentation
on the subject: https://help.github.com/en/articles/autolinked-references-and-urls#issues-and-pull-requests
-->

#### Related Issues

- None

#### Possible Duplicates

- None

## Environment & Settings
<!-- Include the contents of celery --version below -->
**Celery version**:
5.4.0
<!-- Include the output of celery -A proj report below -->
<details>
<summary><b><code>celery report</code> Output:</b></summary>
<p>

```
```

</p>
</details>

# Steps to Reproduce

## Required Dependencies
<!-- Please fill the required dependencies to reproduce this issue -->
- **Minimal Python Version**: N/A or Unknown
- **Minimal Celery Version**:5.4.0
- **Minimal Kombu Version**: N/A or Unknown
- **Minimal Broker Version**: N/A or Unknown
- **Minimal Result Backend Version**: N/A or Unknown
- **Minimal OS and/or Kernel Version**: N/A or Unknown
- **Minimal Broker Client Version**: N/A or Unknown
- **Minimal Result Backend Client Version**: N/A or Unknown

### Python Packages
<!-- Please fill the contents of pip freeze below -->
<details>
<summary><b><code>pip freeze</code> Output:</b></summary>
<p>

```
```

</p>
</details>

### Other Dependencies
<!--
Please provide system dependencies, configuration files
and other dependency information if applicable
-->
<details>
<p>
N/A
</p>
</details>

## Minimally Reproducible Test Case
<!--
Please provide a reproducible test case.
Refer to the Reporting Bugs section in our contribution guide.

We prefer submitting test cases in the form of a PR to our integration test suite.
If you can provide one, please mention the PR number below.
If not, please attach the most minimal code example required to reproduce the issue below.
If the test case is too large, please include a link to a gist or a repository below.

Alternatively, the pytest-celery plugin can be used to create standalone reproduction scripts
that can be added to this report. See the pytest-celery documentation for more information at
pytest-celery.readthedocs.io
-->

<details>
<p>

```python
```

</p>
</details>

# Expected Behavior
<!-- Describe in detail what you expect to happen -->
Return the correct task uuid like before the release 5.4.0
# Actual Behavior
<!--
Describe in detail what actually happened.
Please include a backtrace and surround it with triple backticks (```).
In addition, include the Celery daemon logs, the broker logs,
the result backend logs and system logs below if they will help us debug
the issue.
-->
## Issue ⚠️ 

`AttributeError: 'NoneType' object has no attribute 'id'

```python
@app.task
def example_task(id):
       try:
            obj = Model.objects.get(id=id)
            obj.number += 1
            obj.save(update_fields=["number"])
       except ObjectDoesNotExist as e:
            return "str(e)"


task = example_task.delay_on_commit(example_model_instance.id)
example_model_instance.task_uuid = task.id
```

Literally basic minimal task 😄 , but the `delay_on_commit` not return the task uuid for `AsyncResult`, and in my case django raise excpetion (rightly) AttributeError -> maybe the task with `delay_on_commit` return `None`?

docs:
```python
class DjangoTask(Task):
    """
    Extend the base :class:`~celery.app.task.Task` for Django.

    Provide a nicer API to trigger tasks at the end of the DB transaction.
    """

    def delay_on_commit(self, *args, **kwargs):
        """Call :meth:`~celery.app.task.Task.delay` with Django's ``on_commit()``."""
        return transaction.on_commit(functools.partial(self.delay, *args, **kwargs))

    def apply_async_on_commit(self, *args, **kwargs):
        """Call :meth:`~celery.app.task.Task.apply_async` with Django's ``on_commit()``."""
        return transaction.on_commit(functools.partial(self.apply_async, *args, **kwargs))
```


Discussion:
@browniebroke 
> [x] I have tried to reproduce the issue with pytest-celery and added the reproduction script below.

Where is it?
> > [x] I have tried to reproduce the issue with pytest-celery and added the reproduction script below.
> 
> Where is it?

oh no sorry... my fault
I think that django `transaction.on_commit` should return `None`, but this is the source of  django `transaction,py`
[transaction.py](https://github.com/django/django/blob/main/django/db/transaction.py)

if use the decorator instead the on_commit function?
delay and apply_async like before but:

```python
@transaction.atomic
def delay(.....)

@transaction.atomic
def apply_async(.....)
```
Yes, that's expected behaviour, `transaction.on_commit(...)` delays the execution of the wrapped function to a later stage, when the Django DB transaction finishes, so you won't get a task UUID until then. If you need access to the task UUID, you should use `delay` or `apply_async`, NOT the `..._on_commit` variants. While I didn't forsee this issue, I'm glad we picked entirely new method names, which means there is an escape hatch for you. We should document this caveat better, though.

Your solution to use `@transaction.atomic` decorator does not do the same thing. It will open either a new transaction, or [a savepoint (if you're already in a transaction)](https://docs.djangoproject.com/en/5.0/topics/db/transactions/#savepoints), which may or may not have the same effect. Consider this case, for example:

```python
def create_user(request):
    with transaction.atomic():
        user = User.objects.create(first_name="bob", last_name="doe")
        send_welcome_email.delay_on_commit(user.pk)
        # later in the transaction, do something that cause the transaction to be rolledback
        # e.g. set the email to a value that already exists, raising IntegrityError
        user.email = "bob.doe@example.com"
        user.save() # <- raises IntegrityError, and rollback the whole transaction
```

When the whole transaction is rolled back, the whole operation failed, and user is NOT created in the DB, so we shouldn't send the welcome email. Since the transaction wasn't commited, the task never triggered (and no UUID was generated).

Now if we replace the `delay_on_commit` by a call to `delay` with the `@transaction.atomic`, then the task will be queued in a nested transaction, and we will try to send welcome email (although that will probably crash, since the user wasn't actually persistend in the DB).
I confirm that `delay_on_commit` return None:

```python
In [9]: @app.task
   ...: def sum_number(n1, n2):
   ...:     print(n1+n2)
   ...: 

In [10]: task = sum_number.delay(1,2)

In [11]: task
Out[11]: <AsyncResult: 3a4138ec-35d7-4d46-808f-3f73a1dfee53>

In [12]: task = sum_number.delay_on_commit(1,2)

In [13]: task

In [14]: assert task is None

In [15]: task is None
Out[15]: True
```
> Yes, that's expected behaviour, `transaction.on_commit(...)` delays the execution of the wrapped function to a later stage, when the Django DB transaction finishes, so you won't get a task UUID until then. If you need access to the task UUID, you should use `delay` or `apply_async`, NOT the `..._on_commit` variants. While I didn't forsee this issue, I'm glad we picked entirely new method names, which means there is an escape hatch for you. We should document this caveat better, though.
> 
> Your solution to use `@transaction.atomic` decorator does not do the same thing. It will open either a new transaction, or [a savepoint (if you're already in a transaction)](https://docs.djangoproject.com/en/5.0/topics/db/transactions/#savepoints), which may or may not have the same effect. Consider this case, for example:
> 
> ```python
> def create_user(request):
>     with transaction.atomic():
>         user = User.objects.create(first_name="bob", last_name="doe")
>         send_welcome_email.delay_on_commit(user.pk)
>         # later in the transaction, do something that cause the transaction to be rolledback
>         # e.g. set the email to a value that already exists, raising IntegrityError
>         user.email = "bob.doe@example.com"
>         user.save() # <- raises IntegrityError, and rollback the whole transaction
> ```
> 
> When the whole transaction is rolled back, the whole operation failed, and user is NOT created in the DB, so we shouldn't send the welcome email. Since the transaction wasn't commited, the task never triggered (and no UUID was generated).
> 
> Now if we replace the `delay_on_commit` by `call to`delay`with the`@transaction.atomic`, then the task will be queued in a nested transaction, and we will try to send welcome email (although that will probably crash, since the user wasn't actually persistend in the DB).

Ohh correctly...so if I need the task UUID just use mandatory the `delay` or `apply_async` call task method, correctly?
> so if I need the task UUID just use mandatory the `delay` or `apply_async` call task method, correctly?

Correct, yes
we can develop the way to wait for the end of the transaction and return the task UUID?
or it's a bad idea?
That kind of defeat the whole purpose of the `on_commit` utility, which is a mechanism to delay execution and move on without waiting. Here is the source: 

https://github.com/django/django/blob/ec8552417df51df8482df61b8ad78a7002634011/django/db/transaction.py#L129-L134

Which essentially calls this:

https://github.com/django/django/blob/ec8552417df51df8482df61b8ad78a7002634011/django/db/backends/base/base.py#L727-L750

The function is NOT executed at this point, it's basically added to a list of callbacks (line 732):

```python
self.run_on_commit.append((set(self.savepoint_ids), func, robust))
```

This list is read and executed in `run_and_clear_commit_hooks` (defined below):

https://github.com/django/django/blob/ec8552417df51df8482df61b8ad78a7002634011/django/db/backends/base/base.py#L752C9-L769

That may happen in a completely different context that where you trigger the task.
Let me know if anything is unclear, it would be good to incorporate explanations to lift any doubts you may have in the Celery docs...
> That kind of defeat the whole purpose of the `on_commit` utility, which is a mechanism to delay execution and move on without waiting. Here is the source:
> 
> https://github.com/django/django/blob/ec8552417df51df8482df61b8ad78a7002634011/django/db/transaction.py#L129-L134
> 
> Which essentially calls this:
> 
> https://github.com/django/django/blob/ec8552417df51df8482df61b8ad78a7002634011/django/db/backends/base/base.py#L727-L750
> 
> The function is NOT executed at this point, it's basically added to a list of callbacks (line 732):
> 
> ```python
> self.run_on_commit.append((set(self.savepoint_ids), func, robust))
> ```
> 
> This list is read and executed in `run_and_clear_commit_hooks` (defined below):
> 
> https://github.com/django/django/blob/ec8552417df51df8482df61b8ad78a7002634011/django/db/backends/base/base.py#L752C9-L769
> 
> That may happen in a completely different context that where you trigger the task.

So... it's a way for not waiting the finish of the transaction, correctly?

In this `while`, iterate and call `func` to execute for every function to execute on_commit
https://github.com/django/django/blob/ec8552417df51df8482df61b8ad78a7002634011/django/db/backends/base/base.py#L756-L767

Resolving patch:
diff --git a/celery/contrib/django/task.py b/celery/contrib/django/task.py
--- a/celery/contrib/django/task.py
+++ b/celery/contrib/django/task.py
@@ -12,10 +12,10 @@ class DjangoTask(Task):
     Provide a nicer API to trigger tasks at the end of the DB transaction.
     """
 
-    def delay_on_commit(self, *args, **kwargs):
+    def delay_on_commit(self, *args, **kwargs) -> None:
         """Call :meth:`~celery.app.task.Task.delay` with Django's ``on_commit()``."""
-        return transaction.on_commit(functools.partial(self.delay, *args, **kwargs))
+        transaction.on_commit(functools.partial(self.delay, *args, **kwargs))
 
-    def apply_async_on_commit(self, *args, **kwargs):
+    def apply_async_on_commit(self, *args, **kwargs) -> None:
         """Call :meth:`~celery.app.task.Task.apply_async` with Django's ``on_commit()``."""
-        return transaction.on_commit(functools.partial(self.apply_async, *args, **kwargs))
+        transaction.on_commit(functools.partial(self.apply_async, *args, **kwargs))


Test patch:
diff --git a/t/unit/contrib/django/test_task.py b/t/unit/contrib/django/test_task.py
--- a/t/unit/contrib/django/test_task.py
+++ b/t/unit/contrib/django/test_task.py
@@ -25,8 +25,8 @@ def on_commit(self):
 
     def test_delay_on_commit(self, task_instance, on_commit):
         result = task_instance.delay_on_commit()
-        assert result is not None
+        assert result is None
 
     def test_apply_async_on_commit(self, task_instance, on_commit):
         result = task_instance.apply_async_on_commit()
-        assert result is not None
+        assert result is None