Repository: tornadoweb/tornado

Issue:
Test failures with Python 3.12 b1 & b2
Hello,

I'd like to ask for help with tornado and its compatibility with Python 3.12. Up until the 7th alpha tornado was building fine. I started to see the following failures with the 1st beta. You can see the failure also in CI of this PR https://github.com/tornadoweb/tornado/pull/3277.

Thank you.

~~~~
======================================================================
FAIL: test_chunked (tornado.test.httpclient_test.HTTPClientCommonTestCase.test_chunked)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/tornado-6.3.2/tornado/testing.py", line 102, in __call__
    result = self.orig_method(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/tornado-6.3.2/tornado/test/httpclient_test.py", line 225, in test_chunked
    self.assertEqual(chunks, [b"asdf", b"qwer"])
AssertionError: Lists differ: [b'asdfqwer'] != [b'asdf', b'qwer']

First differing element 0:
b'asdfqwer'
b'asdf'

Second list contains 1 additional elements.
First extra element 1:
b'qwer'

- [b'asdfqwer']
+ [b'asdf', b'qwer']
?        +++++


======================================================================
FAIL: test_gzip (tornado.test.httpclient_test.HTTPClientCommonTestCase.test_gzip)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/tornado-6.3.2/tornado/testing.py", line 102, in __call__
    result = self.orig_method(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/tornado-6.3.2/tornado/test/httpclient_test.py", line 438, in test_gzip
    self.assertEqual(len(response.body), 34)
AssertionError: 28 != 34

======================================================================
FAIL: test_header_callback (tornado.test.httpclient_test.HTTPClientCommonTestCase.test_header_callback)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/tornado-6.3.2/tornado/testing.py", line 102, in __call__
    result = self.orig_method(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/tornado-6.3.2/tornado/test/httpclient_test.py", line 484, in test_header_callback
    self.assertEqual(chunks, [b"asdf", b"qwer"])
AssertionError: Lists differ: [b'asdfqwer'] != [b'asdf', b'qwer']

First differing element 0:
b'asdfqwer'
b'asdf'

Second list contains 1 additional elements.
First extra element 1:
b'qwer'

- [b'asdfqwer']
+ [b'asdf', b'qwer']
?        +++++


======================================================================
FAIL: test_chunked (tornado.test.simple_httpclient_test.SimpleHTTPClientCommonTestCase.test_chunked)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/tornado-6.3.2/tornado/testing.py", line 102, in __call__
    result = self.orig_method(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/tornado-6.3.2/tornado/test/httpclient_test.py", line 225, in test_chunked
    self.assertEqual(chunks, [b"asdf", b"qwer"])
AssertionError: Lists differ: [b'asdfqwer'] != [b'asdf', b'qwer']

First differing element 0:
b'asdfqwer'
b'asdf'

Second list contains 1 additional elements.
First extra element 1:
b'qwer'

- [b'asdfqwer']
+ [b'asdf', b'qwer']
?        +++++


======================================================================
FAIL: test_gzip (tornado.test.simple_httpclient_test.SimpleHTTPClientCommonTestCase.test_gzip)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/tornado-6.3.2/tornado/testing.py", line 102, in __call__
    result = self.orig_method(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/tornado-6.3.2/tornado/test/httpclient_test.py", line 438, in test_gzip
    self.assertEqual(len(response.body), 34)
AssertionError: 28 != 34

======================================================================
FAIL: test_header_callback (tornado.test.simple_httpclient_test.SimpleHTTPClientCommonTestCase.test_header_callback)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILD/tornado-6.3.2/tornado/testing.py", line 102, in __call__
    result = self.orig_method(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builddir/build/BUILD/tornado-6.3.2/tornado/test/httpclient_test.py", line 484, in test_header_callback
    self.assertEqual(chunks, [b"asdf", b"qwer"])
AssertionError: Lists differ: [b'asdfqwer'] != [b'asdf', b'qwer']

First differing element 0:
b'asdfqwer'
b'asdf'

Second list contains 1 additional elements.
First extra element 1:
b'qwer'

- [b'asdfqwer']
+ [b'asdf', b'qwer']
?        +++++


----------------------------------------------------------------------
Ran 1189 tests in 17.011s

FAILED (failures=6, skipped=66)
~~~~
build: Upgrade tox to support python 3.12



Discussion:
This is a bug introduced by a change to the python standard library's gzip module: python/cpython#105808. Hopefully it'll get fixed in an upcoming beta. 

There are also other things that broke with the 3.12 betas: we need to upgrade tox, and there are some deprecation warnings in datetime that we have to work around. I have patches for those issues but there's no clean way to work around the GzipFile bug so I'm going to disable 3.12 in CI until the fix comes out. 


Resolving patch:
diff --git a/tornado/httputil.py b/tornado/httputil.py
--- a/tornado/httputil.py
+++ b/tornado/httputil.py
@@ -856,7 +856,8 @@ def format_timestamp(
 
     The argument may be a numeric timestamp as returned by `time.time`,
     a time tuple as returned by `time.gmtime`, or a `datetime.datetime`
-    object.
+    object. Naive `datetime.datetime` objects are assumed to represent
+    UTC; aware objects are converted to UTC before formatting.
 
     >>> format_timestamp(1359312200)
     'Sun, 27 Jan 2013 18:43:20 GMT'
diff --git a/tornado/locale.py b/tornado/locale.py
--- a/tornado/locale.py
+++ b/tornado/locale.py
@@ -333,7 +333,7 @@ def format_date(
         shorter: bool = False,
         full_format: bool = False,
     ) -> str:
-        """Formats the given date (which should be GMT).
+        """Formats the given date.
 
         By default, we return a relative time (e.g., "2 minutes ago"). You
         can return an absolute date string with ``relative=False``.
@@ -343,10 +343,16 @@ def format_date(
 
         This method is primarily intended for dates in the past.
         For dates in the future, we fall back to full format.
+
+        .. versionchanged:: 6.4
+           Aware `datetime.datetime` objects are now supported (naive
+           datetimes are still assumed to be UTC).
         """
         if isinstance(date, (int, float)):
-            date = datetime.datetime.utcfromtimestamp(date)
-        now = datetime.datetime.utcnow()
+            date = datetime.datetime.fromtimestamp(date, datetime.timezone.utc)
+        if date.tzinfo is None:
+            date = date.replace(tzinfo=datetime.timezone.utc)
+        now = datetime.datetime.now(datetime.timezone.utc)
         if date > now:
             if relative and (date - now).seconds < 60:
                 # Due to click skew, things are some things slightly
diff --git a/tornado/web.py b/tornado/web.py
--- a/tornado/web.py
+++ b/tornado/web.py
@@ -647,7 +647,9 @@ def set_cookie(
         if domain:
             morsel["domain"] = domain
         if expires_days is not None and not expires:
-            expires = datetime.datetime.utcnow() + datetime.timedelta(days=expires_days)
+            expires = datetime.datetime.now(datetime.timezone.utc) + datetime.timedelta(
+                days=expires_days
+            )
         if expires:
             morsel["expires"] = httputil.format_timestamp(expires)
         if path:
@@ -698,7 +700,9 @@ def clear_cookie(self, name: str, **kwargs: Any) -> None:
                 raise TypeError(
                     f"clear_cookie() got an unexpected keyword argument '{excluded_arg}'"
                 )
-        expires = datetime.datetime.utcnow() - datetime.timedelta(days=365)
+        expires = datetime.datetime.now(datetime.timezone.utc) - datetime.timedelta(
+            days=365
+        )
         self.set_cookie(name, value="", expires=expires, **kwargs)
 
     def clear_all_cookies(self, **kwargs: Any) -> None:
@@ -2812,12 +2816,12 @@ def should_return_304(self) -> bool:
         # content has not been modified
         ims_value = self.request.headers.get("If-Modified-Since")
         if ims_value is not None:
-            date_tuple = email.utils.parsedate(ims_value)
-            if date_tuple is not None:
-                if_since = datetime.datetime(*date_tuple[:6])
-                assert self.modified is not None
-                if if_since >= self.modified:
-                    return True
+            if_since = email.utils.parsedate_to_datetime(ims_value)
+            if if_since.tzinfo is None:
+                if_since = if_since.replace(tzinfo=datetime.timezone.utc)
+            assert self.modified is not None
+            if if_since >= self.modified:
+                return True
 
         return False
 
@@ -2981,6 +2985,10 @@ def get_modified_time(self) -> Optional[datetime.datetime]:
         object or None.
 
         .. versionadded:: 3.1
+
+        .. versionchanged:: 6.4
+           Now returns an aware datetime object instead of a naive one.
+           Subclasses that override this method may return either kind.
         """
         stat_result = self._stat()
         # NOTE: Historically, this used stat_result[stat.ST_MTIME],
@@ -2991,7 +2999,9 @@ def get_modified_time(self) -> Optional[datetime.datetime]:
         # consistency with the past (and because we have a unit test
         # that relies on this), we truncate the float here, although
         # I'm not sure that's the right thing to do.
-        modified = datetime.datetime.utcfromtimestamp(int(stat_result.st_mtime))
+        modified = datetime.datetime.fromtimestamp(
+            int(stat_result.st_mtime), datetime.timezone.utc
+        )
         return modified
 
     def get_content_type(self) -> str:


Test patch:
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -51,9 +51,8 @@ jobs:
             tox_env: py311-full
           - python: '3.11.0'
             tox_env: py311-full
-          # 3.12 is disabled until https://github.com/python/cpython/issues/105808 is fixed
-          #- python: '3.12.0-alpha - 3.12'
-          #  tox_env: py312-full
+          - python: '3.12.0-beta.3 - 3.12'
+            tox_env: py312-full
           - python: 'pypy-3.8'
             # Pypy is a lot slower due to jit warmup costs, so don't run the
             # "full" test config there.
diff --git a/tornado/test/httpclient_test.py b/tornado/test/httpclient_test.py
--- a/tornado/test/httpclient_test.py
+++ b/tornado/test/httpclient_test.py
@@ -28,7 +28,7 @@
 from tornado.log import gen_log, app_log
 from tornado import netutil
 from tornado.testing import AsyncHTTPTestCase, bind_unused_port, gen_test, ExpectLog
-from tornado.test.util import skipOnTravis
+from tornado.test.util import skipOnTravis, ignore_deprecation
 from tornado.web import Application, RequestHandler, url
 from tornado.httputil import format_timestamp, HTTPHeaders
 
@@ -887,7 +887,15 @@ def test_body_setter(self):
         self.assertEqual(request.body, utf8("foo"))
 
     def test_if_modified_since(self):
-        http_date = datetime.datetime.utcnow()
+        http_date = datetime.datetime.now(datetime.timezone.utc)
+        request = HTTPRequest("http://example.com", if_modified_since=http_date)
+        self.assertEqual(
+            request.headers, {"If-Modified-Since": format_timestamp(http_date)}
+        )
+
+    def test_if_modified_since_naive_deprecated(self):
+        with ignore_deprecation():
+            http_date = datetime.datetime.utcnow()
         request = HTTPRequest("http://example.com", if_modified_since=http_date)
         self.assertEqual(
             request.headers, {"If-Modified-Since": format_timestamp(http_date)}
diff --git a/tornado/test/httputil_test.py b/tornado/test/httputil_test.py
--- a/tornado/test/httputil_test.py
+++ b/tornado/test/httputil_test.py
@@ -13,6 +13,7 @@
 from tornado.escape import utf8, native_str
 from tornado.log import gen_log
 from tornado.testing import ExpectLog
+from tornado.test.util import ignore_deprecation
 
 import copy
 import datetime
@@ -412,8 +413,29 @@ def test_time_tuple(self):
         self.assertEqual(9, len(tup))
         self.check(tup)
 
-    def test_datetime(self):
-        self.check(datetime.datetime.utcfromtimestamp(self.TIMESTAMP))
+    def test_utc_naive_datetime(self):
+        self.check(
+            datetime.datetime.fromtimestamp(
+                self.TIMESTAMP, datetime.timezone.utc
+            ).replace(tzinfo=None)
+        )
+
+    def test_utc_naive_datetime_deprecated(self):
+        with ignore_deprecation():
+            self.check(datetime.datetime.utcfromtimestamp(self.TIMESTAMP))
+
+    def test_utc_aware_datetime(self):
+        self.check(
+            datetime.datetime.fromtimestamp(self.TIMESTAMP, datetime.timezone.utc)
+        )
+
+    def test_other_aware_datetime(self):
+        # Other timezones are ignored; the timezone is always printed as GMT
+        self.check(
+            datetime.datetime.fromtimestamp(
+                self.TIMESTAMP, datetime.timezone(datetime.timedelta(hours=-4))
+            )
+        )
 
 
 # HTTPServerRequest is mainly tested incidentally to the server itself,
diff --git a/tornado/test/locale_test.py b/tornado/test/locale_test.py
--- a/tornado/test/locale_test.py
+++ b/tornado/test/locale_test.py
@@ -91,45 +91,55 @@ def test_format_date(self):
             locale.format_date(date, full_format=True), "April 28, 2013 at 6:35 pm"
         )
 
-        now = datetime.datetime.utcnow()
-
-        self.assertEqual(
-            locale.format_date(now - datetime.timedelta(seconds=2), full_format=False),
-            "2 seconds ago",
-        )
-        self.assertEqual(
-            locale.format_date(now - datetime.timedelta(minutes=2), full_format=False),
-            "2 minutes ago",
-        )
-        self.assertEqual(
-            locale.format_date(now - datetime.timedelta(hours=2), full_format=False),
-            "2 hours ago",
-        )
-
-        self.assertEqual(
-            locale.format_date(
-                now - datetime.timedelta(days=1), full_format=False, shorter=True
-            ),
-            "yesterday",
-        )
-
-        date = now - datetime.timedelta(days=2)
-        self.assertEqual(
-            locale.format_date(date, full_format=False, shorter=True),
-            locale._weekdays[date.weekday()],
-        )
-
-        date = now - datetime.timedelta(days=300)
-        self.assertEqual(
-            locale.format_date(date, full_format=False, shorter=True),
-            "%s %d" % (locale._months[date.month - 1], date.day),
-        )
-
-        date = now - datetime.timedelta(days=500)
-        self.assertEqual(
-            locale.format_date(date, full_format=False, shorter=True),
-            "%s %d, %d" % (locale._months[date.month - 1], date.day, date.year),
-        )
+        aware_dt = datetime.datetime.now(datetime.timezone.utc)
+        naive_dt = aware_dt.replace(tzinfo=None)
+        for name, now in {"aware": aware_dt, "naive": naive_dt}.items():
+            with self.subTest(dt=name):
+                self.assertEqual(
+                    locale.format_date(
+                        now - datetime.timedelta(seconds=2), full_format=False
+                    ),
+                    "2 seconds ago",
+                )
+                self.assertEqual(
+                    locale.format_date(
+                        now - datetime.timedelta(minutes=2), full_format=False
+                    ),
+                    "2 minutes ago",
+                )
+                self.assertEqual(
+                    locale.format_date(
+                        now - datetime.timedelta(hours=2), full_format=False
+                    ),
+                    "2 hours ago",
+                )
+
+                self.assertEqual(
+                    locale.format_date(
+                        now - datetime.timedelta(days=1),
+                        full_format=False,
+                        shorter=True,
+                    ),
+                    "yesterday",
+                )
+
+                date = now - datetime.timedelta(days=2)
+                self.assertEqual(
+                    locale.format_date(date, full_format=False, shorter=True),
+                    locale._weekdays[date.weekday()],
+                )
+
+                date = now - datetime.timedelta(days=300)
+                self.assertEqual(
+                    locale.format_date(date, full_format=False, shorter=True),
+                    "%s %d" % (locale._months[date.month - 1], date.day),
+                )
+
+                date = now - datetime.timedelta(days=500)
+                self.assertEqual(
+                    locale.format_date(date, full_format=False, shorter=True),
+                    "%s %d, %d" % (locale._months[date.month - 1], date.day, date.year),
+                )
 
     def test_friendly_number(self):
         locale = tornado.locale.get("en_US")
diff --git a/tornado/test/web_test.py b/tornado/test/web_test.py
--- a/tornado/test/web_test.py
+++ b/tornado/test/web_test.py
@@ -404,10 +404,10 @@ def test_set_cookie_expires_days(self):
         match = re.match("foo=bar; expires=(?P<expires>.+); Path=/", header)
         assert match is not None
 
-        expires = datetime.datetime.utcnow() + datetime.timedelta(days=10)
-        parsed = email.utils.parsedate(match.groupdict()["expires"])
-        assert parsed is not None
-        header_expires = datetime.datetime(*parsed[:6])
+        expires = datetime.datetime.now(datetime.timezone.utc) + datetime.timedelta(
+            days=10
+        )
+        header_expires = email.utils.parsedate_to_datetime(match.groupdict()["expires"])
         self.assertTrue(abs((expires - header_expires).total_seconds()) < 10)
 
     def test_set_cookie_false_flags(self):
@@ -1697,11 +1697,10 @@ def get(self):
 
     def test_date_header(self):
         response = self.fetch("/")
-        parsed = email.utils.parsedate(response.headers["Date"])
-        assert parsed is not None
-        header_date = datetime.datetime(*parsed[:6])
+        header_date = email.utils.parsedate_to_datetime(response.headers["Date"])
         self.assertTrue(
-            header_date - datetime.datetime.utcnow() < datetime.timedelta(seconds=2)
+            header_date - datetime.datetime.now(datetime.timezone.utc)
+            < datetime.timedelta(seconds=2)
         )
 
 
@@ -3010,10 +3009,12 @@ def test_xsrf_httponly(self):
         match = re.match(".*; expires=(?P<expires>.+);.*", header)
         assert match is not None
 
-        expires = datetime.datetime.utcnow() + datetime.timedelta(days=2)
-        parsed = email.utils.parsedate(match.groupdict()["expires"])
-        assert parsed is not None
-        header_expires = datetime.datetime(*parsed[:6])
+        expires = datetime.datetime.now(datetime.timezone.utc) + datetime.timedelta(
+            days=2
+        )
+        header_expires = email.utils.parsedate_to_datetime(match.groupdict()["expires"])
+        if header_expires.tzinfo is None:
+            header_expires = header_expires.replace(tzinfo=datetime.timezone.utc)
         self.assertTrue(abs((expires - header_expires).total_seconds()) < 10)