Repository: celery/celery (Python)

Issue:
ETA field on task is not correctly parsed in celery>=5.3.0 and python<3.11
<!--
Please fill this template entirely and do not erase parts of it.
We reserve the right to close without a response
bug reports which are incomplete.
-->
# Checklist
<!--
To check an item on the list replace [ ] with [x].
-->
- [ ] I have verified that the issue exists against the `main` branch of Celery.
- [ ] This has already been asked to the [discussions forum](https://github.com/celery/celery/discussions) first.
- [x] I have read the relevant section in the
  [contribution guide](https://docs.celeryq.dev/en/main/contributing.html#other-bugs)
  on reporting bugs.
- [x] I have checked the [issues list](https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22)
  for similar or identical bug reports.
- [x] I have checked the [pull requests list](https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22)
  for existing proposed fixes.
- [x] I have checked the [commit log](https://github.com/celery/celery/commits/main)
  to find out if the bug was already fixed in the main branch.
- [x] I have included all related issues and possible duplicate issues
  in this issue (If there are none, check this box anyway).

## Mandatory Debugging Information

- [ ] I have included the output of ``celery -A proj report`` in the issue.
    (if you are not able to do this, then at least specify the Celery
     version affected).
- [ ] I have verified that the issue exists against the `main` branch of Celery.
- [ ] I have included the contents of ``pip freeze`` in the issue.
- [ ] I have included all the versions of all the external dependencies required
  to reproduce this bug.

## Optional Debugging Information
<!--
Try some of the below if you think they are relevant.
It will help us figure out the scope of the bug and how many users it affects.
-->
- [x] I have tried reproducing the issue on more than one Python version
  and/or implementation.
- [ ] I have tried reproducing the issue on more than one message broker and/or
  result backend.
- [ ] I have tried reproducing the issue on more than one version of the message
  broker and/or result backend.
- [ ] I have tried reproducing the issue on more than one operating system.
- [ ] I have tried reproducing the issue on more than one workers pool.
- [ ] I have tried reproducing the issue with autoscaling, retries,
  ETA/Countdown & rate limits disabled.
- [x] I have tried reproducing the issue after downgrading
  and/or upgrading Celery and its dependencies.

## Related Issues and Possible Duplicates
<!--
Please make sure to search and mention any related issues
or possible duplicates to this issue as requested by the checklist above.

This may or may not include issues in other repositories that the Celery project
maintains or other repositories that are dependencies of Celery.

If you don't know how to mention issues, please refer to Github's documentation
on the subject: https://help.github.com/en/articles/autolinked-references-and-urls#issues-and-pull-requests
-->

#### Related Issues

- None

#### Possible Duplicates

- None

## Environment & Settings
<!-- Include the contents of celery --version below -->
**Celery version**:
<!-- Include the output of celery -A proj report below -->
<details>
<summary><b><code>celery report</code> Output:</b></summary>
<p>

```
```

</p>
</details>

# Steps to Reproduce

## Required Dependencies
<!-- Please fill the required dependencies to reproduce this issue -->
- **Minimal Python Version**: N/A or Unknown
- **Minimal Celery Version**: 5.3.0
- **Minimal Kombu Version**: N/A or Unknown
- **Minimal Broker Version**: N/A or Unknown
- **Minimal Result Backend Version**: N/A or Unknown
- **Minimal OS and/or Kernel Version**: N/A or Unknown
- **Minimal Broker Client Version**: N/A or Unknown
- **Minimal Result Backend Client Version**: N/A or Unknown

### Python Packages
<!-- Please fill the contents of pip freeze below -->
<details>
<summary><b><code>pip freeze</code> Output:</b></summary>
<p>

```
```

</p>
</details>

### Other Dependencies
<!--
Please provide system dependencies, configuration files
and other dependency information if applicable
-->
<details>
<p>
N/A
</p>
</details>

## Minimally Reproducible Test Case
<!--
Please provide a reproducible test case.
Refer to the Reporting Bugs section in our contribution guide.

We prefer submitting test cases in the form of a PR to our integration test suite.
If you can provide one, please mention the PR number below.
If not, please attach the most minimal code example required to reproduce the issue below.
If the test case is too large, please include a link to a gist or a repository below.
-->

<details>
<p>

```python
```

</p>
</details>

# Expected Behavior
<!-- Describe in detail what you expect to happen -->

When adding a task to the queue with a valid iso8601 ETA field set (for example) `'2023-09-12T15:39:16Z`, celery should accept the task and execute it correspondingly.

# Actual Behavior
<!--
Describe in detail what actually happened.
Please include a backtrace and surround it with triple backticks (```).
In addition, include the Celery daemon logs, the broker logs,
the result backend logs and system logs below if they will help us debug
the issue.
-->

```
compose-worker_fast-5  | [2023-09-12 15:39:16,095: ERROR/MainProcess] Received invalid task message: invalid ETA value '
2023-09-12T15:39:16Z': Invalid isoformat string: '2023-09-12T15:39:16Z'
compose-worker_fast-5  | The message has been ignored and discarded.
compose-worker_fast-5  |
compose-worker_fast-5  | Please ensure your message conforms to the task
compose-worker_fast-5  | message protocol as described here:
compose-worker_fast-5  | https://docs.celeryq.dev/en/latest/internals/protocol.html
compose-worker_fast-5  |
compose-worker_fast-5  | The full contents of the message body was:
compose-worker_fast-5  | {'id': '', 'task': 'tasks.notification.alert.send_notification_alert_mail', 'args': [3022248],
'kwargs': {}, 'retries': 0, 'eta': '2023-09-12T15:39:16Z'} (142b)
compose-worker_fast-5  | Traceback (most recent call last):
compose-worker_fast-5  |   File "/home/celery/venv/lib/python3.10/site-packages/celery/worker/request.py", line 135, in
__init__
compose-worker_fast-5  |     eta = maybe_iso8601(eta)
compose-worker_fast-5  |   File "/home/celery/venv/lib/python3.10/site-packages/celery/utils/time.py", line 291, in mayb
e_iso8601
compose-worker_fast-5  |     return datetime.fromisoformat(dt)
compose-worker_fast-5  | ValueError: Invalid isoformat string: '2023-09-12T15:39:16Z'
compose-worker_fast-5  |
compose-worker_fast-5  | During handling of the above exception, another exception occurred:
compose-worker_fast-5  |
compose-worker_fast-5  | Traceback (most recent call last):
compose-worker_fast-5  |   File "/home/celery/venv/lib/python3.10/site-packages/celery/worker/consumer/consumer.py", lin
e 668, in on_task_received
compose-worker_fast-5  |     strategy(
compose-worker_fast-5  |   File "/home/celery/venv/lib/python3.10/site-packages/celery/worker/strategy.py", line 146, in
 task_message_handler
compose-worker_fast-5  |     req = Req(
compose-worker_fast-5  |   File "/home/celery/venv/lib/python3.10/site-packages/celery/worker/request.py", line 137, in
__init__
compose-worker_fast-5  |     raise InvalidTaskError(
compose-worker_fast-5  | celery.exceptions.InvalidTaskError: invalid ETA value '2023-09-12T15:39:16Z': Invalid isoformat
 string: '2023-09-12T15:39:16Z'

```

The problem is that in 5.3.0[ this change](https://github.com/celery/celery/pull/8098) switched to python's native `datetime.datetime.isoformat` to parse iso8601 dates in `eta` field. However, python<3.11 implemented this incorrectly (which is probably why you implemented your own parser function in the first place). When running celery 5.3.0 in python 3.11, this will work, but when running in 3.10 or lower the above ValueError will be thrown.

The fix is probably as simple as checking your python version and using your own (deprecated) function in python 3.10 or lower and using python's native datetime method in python 3.11 or higher


Discussion:
Temporary fix is rolling back to the celery version before 5.3.0
thanks for reporting the issue
It's the Z timezone portion, that's what older Python doesn't handle.  Replacing Z with +00:00 produces an equivalent aware datetime.  

Probably better to handle that internally?  So maybe change back to the util function, but use the built-in there?
It's a valid iso8601 date, which according to the celery docs is allowed. It was previously allowed until 5.3.0 when the parsing was switched from the internal method to the datetime.datetime.isoformat which implemented this correctly in python 3.11, but not in python 3.10 and below.

Seems to me that this is a breaking change for everyone using python 3.10 with celery, so yes internally we could easily fix this by not using `Z` but I think this should be fixed in celery as well, as this is basically an API break.
> Probably better to handle that internally

As I said...
thanks for the report and fix & investigation!!
Thanks for the quick fix!



Patch:
diff --git a/.github/workflows/python-package.yml b/.github/workflows/python-package.yml
index 2049fe37211..078c5a4fbb5 100644
--- a/.github/workflows/python-package.yml
+++ b/.github/workflows/python-package.yml
@@ -36,7 +36,7 @@ jobs:
               os: "windows-latest"
             - python-version: 'pypy-3.10'
               os: "windows-latest"
-            
+
     steps:
     - name: Install apt packages
       if: startsWith(matrix.os, 'ubuntu-')
diff --git a/celery/app/base.py b/celery/app/base.py
index fb78893ba2d..4846a913bf4 100644
--- a/celery/app/base.py
+++ b/celery/app/base.py
@@ -9,6 +9,7 @@
 from operator import attrgetter
 
 from click.exceptions import Exit
+from dateutil.parser import isoparse
 from kombu import pools
 from kombu.clocks import LamportClock
 from kombu.common import oid_from
@@ -740,7 +741,7 @@ def send_task(self, name, args=None, kwargs=None, countdown=None,
                     expires) - self.now()).total_seconds()
             elif isinstance(expires, str):
                 expires_s = (maybe_make_aware(
-                    datetime.fromisoformat(expires)) - self.now()).total_seconds()
+                    isoparse(expires)) - self.now()).total_seconds()
             else:
                 expires_s = expires
 
diff --git a/celery/result.py b/celery/result.py
index 0c9e0a30f21..065d9ca5158 100644
--- a/celery/result.py
+++ b/celery/result.py
@@ -6,6 +6,7 @@
 from contextlib import contextmanager
 from weakref import proxy
 
+from dateutil.parser import isoparse
 from kombu.utils.objects import cached_property
 from vine import Thenable, barrier, promise
 
@@ -532,7 +533,7 @@ def date_done(self):
         """UTC date and time."""
         date_done = self._get_task_meta().get('date_done')
         if date_done and not isinstance(date_done, datetime.datetime):
-            return datetime.datetime.fromisoformat(date_done)
+            return isoparse(date_done)
         return date_done
 
     @property
diff --git a/celery/utils/iso8601.py b/celery/utils/iso8601.py
index ffe342b40c8..74aff491a69 100644
--- a/celery/utils/iso8601.py
+++ b/celery/utils/iso8601.py
@@ -52,7 +52,7 @@
 
 def parse_iso8601(datestring):
     """Parse and convert ISO-8601 string to datetime."""
-    warn("parse_iso8601", "v5.3", "v6", "datetime.datetime.fromisoformat")
+    warn("parse_iso8601", "v5.3", "v6", "datetime.datetime.fromisoformat or dateutil.parser.isoparse")
     m = ISO8601_REGEX.match(datestring)
     if not m:
         raise ValueError('unable to parse date string %r' % datestring)
diff --git a/celery/utils/time.py b/celery/utils/time.py
index f5329a5e39b..ba94d7951b1 100644
--- a/celery/utils/time.py
+++ b/celery/utils/time.py
@@ -14,6 +14,7 @@
 from typing import Any, Callable
 
 from dateutil import tz as dateutil_tz
+from dateutil.parser import isoparse
 from kombu.utils.functional import reprcall
 from kombu.utils.objects import cached_property
 
@@ -288,7 +289,7 @@ def maybe_iso8601(dt: datetime | str | None) -> None | datetime:
         return
     if isinstance(dt, datetime):
         return dt
-    return datetime.fromisoformat(dt)
+    return isoparse(dt)
 
 
 def is_naive(dt: datetime) -> bool:


Test patch:
diff --git a/t/unit/utils/test_time.py b/t/unit/utils/test_time.py
index 9841f364c5a..80d5db973a1 100644
--- a/t/unit/utils/test_time.py
+++ b/t/unit/utils/test_time.py
@@ -101,6 +101,18 @@ def test_maybe_iso8601_datetime():
     assert maybe_iso8601(now) is now
 
 
+@pytest.mark.parametrize('date_str,expected', [
+    ('2011-11-04T00:05:23', datetime(2011, 11, 4, 0, 5, 23)),
+    ('2011-11-04T00:05:23Z', datetime(2011, 11, 4, 0, 5, 23, tzinfo=_timezone.utc)),
+    ('2011-11-04 00:05:23.283+00:00',
+     datetime(2011, 11, 4, 0, 5, 23, 283000, tzinfo=_timezone.utc)),
+    ('2011-11-04T00:05:23+04:00',
+     datetime(2011, 11, 4, 0, 5, 23,  tzinfo=_timezone(timedelta(seconds=14400)))),
+])
+def test_iso8601_string_datetime(date_str, expected):
+    assert maybe_iso8601(date_str) == expected
+
+
 @pytest.mark.parametrize('arg,expected', [
     (30, timedelta(seconds=30)),
     (30.6, timedelta(seconds=30.6)),