Repository: tornadoweb/tornado
Pull request #3482 (2025-04-22 16:19:42)

Don't define Py_LIMITED_API and don't re-enable GIL under free-threading
- Defining the `Py_LIMITED_API` macro leads to build failures
- Do not re-enable GIL when importing `tornado.speedups` since it's thread-safe.


Patch:
diff --git a/docs/releases.rst b/docs/releases.rst
index 8a0fad4c..1e65c337 100644
--- a/docs/releases.rst
+++ b/docs/releases.rst
@@ -4,6 +4,8 @@ Release notes
 .. toctree::
    :maxdepth: 2
 
+   releases/v6.5.0
+   releases/v6.4.2
    releases/v6.4.1
    releases/v6.4.0
    releases/v6.3.3
diff --git a/docs/releases/v6.4.2.rst b/docs/releases/v6.4.2.rst
new file mode 100644
index 00000000..0dc567d1
--- /dev/null
+++ b/docs/releases/v6.4.2.rst
@@ -0,0 +1,12 @@
+What's new in Tornado 6.4.2
+===========================
+
+Nov 21, 2024
+------------
+
+Security Improvements
+~~~~~~~~~~~~~~~~~~~~~
+
+- Parsing of the cookie header is now much more efficient. The older algorithm sometimes had
+  quadratic performance which allowed for a denial-of-service attack in which the server would spend
+  excessive CPU time parsing cookies and block the event loop. This change fixes CVE-2024-7592.
\ No newline at end of file
diff --git a/docs/releases/v6.5.0.rst b/docs/releases/v6.5.0.rst
new file mode 100644
index 00000000..3a7907af
--- /dev/null
+++ b/docs/releases/v6.5.0.rst
@@ -0,0 +1,91 @@
+What's new in Tornado 6.5.0
+===========================
+
+In progress
+-----------
+
+General Changes
+~~~~~~~~~~~~~~~
+
+- Python 3.14 is now supported. Older versions of Tornado will work on Python 3.14 but may log
+  deprecation warnings.
+- The minimum supported Python version is 3.9.
+
+Deprecation Notices
+~~~~~~~~~~~~~~~~~~~
+
+- Support for ``obs-fold`` continuation lines in HTTP headers is deprecated and will be removed in
+  Tornado 7.0, as is the use of carriage returns without line feeds as header separators.
+- The ``callback`` argument to ``websocket_connect`` is deprecated and will be removed in
+  Tornado 7.0. Note that ``on_message_callback`` is not deprecated.
+- The ``log_message`` and ``args`` attributes of `tornado.web.HTTPError` are deprecated. Use the
+  new ``get_message`` method instead.
+
+Type Annotation Changes
+~~~~~~~~~~~~~~~~~~~~~~~
+
+- `tornado.concurrent.chain_future` is now typed as accepting both asyncio and concurrent Futures.
+- `tornado.gen.multi` and ``multi_future`` now take ``Sequences`` and ``Mappings`` instead of
+  ``List`` and ``Dict``.
+- `tornado.httputil.RequestStartLine` and `.ResponseStartLine` now have type annotations for
+  their attributes.
+- `.HTTPHeaders` now has type annotations for its elements.
+- The ``autoescape`` argument to `tornado.template.BaseLoader` is now marked as optional.
+- ``tornado.routing._RuleList`` is now a ``Sequence`` for more flexibility.
+- ``.RequestHandler.SUPPPORTED_METHODS`` is now typed to support overriding in a subclass.
+- Types for `.RequestHandler.get_body_argument` and ``get_query_argument`` are improved and now
+  match the ``get_argument`` method.
+- `.RequestHandler.get_cookie` now has more accurate types.
+- The return type of `.UIModule.render` may now be either `str` or `bytes`.
+
+``tornado.httputil``
+~~~~~~~~~~~~~~~~~~~~
+
+- Support for ``obs-fold`` continuation lines in HTTP headers is deprecated and will be removed in
+  Tornado 7.0, as is the use of carriage returns without line feeds as header separators.
+- Request start lines may no longer include control characters.
+- Method names containing invalid characters now return error code 400 instead of 405.
+- Header names are now restricted to the set of characters permitted by the RFCs.
+- Control characters are no longer allowed in (incoming) header values.
+- Handling of trailing whitespace in headers has been improved, especially with ``obs-fold``
+  continuation lines.
+- The ``Host`` header is now restricted to the set of characters permitted by the RFCs. It is now an
+  error to send more than one ``Host`` header, or to omit a ``Host`` header for a request that is
+  not using HTTP/1.0.
+
+``tornado.netutil``
+~~~~~~~~~~~~~~~~~~~
+
+- ``.bind_unix_socket`` now supports the Linux abstract namespace.
+
+
+``tornado.platform.twisted``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+- ``TwistedResolver`` has been deleted. It was already deprecated and scheduled for removal
+  in Tornado 7.0, but due to the adoption of RFC 8482 it no longer works for most
+  domain names. This class was primarily intended to provide thread-free non-blocking
+  DNS resolution. If that is still required, ``tornado.platform.caresresolver`` is the
+  next best option, although it has its own limitations which differ from TwistedResolver,
+  and it is also deprecated. Most users should switch to the default resolver, which uses
+  threads.
+
+``tornado.web``
+~~~~~~~~~~~~~~~
+
+- The set of characters allowed in (outgoing) HTTP headers now matches the RFCs. Specifically, tab
+  characters are now allowed and DEL is not.
+- Invalid ``If-Modified-Since`` headers are now ignored instead of causing a 500 error.
+- ``%`` characters in the log message of ``tornado.web.HTTPError`` are no longer doubled when no
+  additional arguments are passed. This matches the behavior of `logging.LogRecord`. A new method
+  ``get_message`` has been added to ``HTTPError`` to allow access to the fully-substituted message;
+  directly accessing ``log_message`` and ``args`` is deprecated.
+
+``tornado.websocket``
+~~~~~~~~~~~~~~~~~~~~~
+
+- Some bugs involving ``ping_interval`` and ``ping_timeout`` have been fixed. Setting the
+  ``ping_timeout`` greater than the ``ping_interval`` is no longer supported. The default
+  ``ping_timeout`` is now equal to the ``ping_interval``. 
+- The ``callback`` argument to ``websocket_connect`` is deprecated and will be removed in
+  Tornado 7.0. Note that ``on_message_callback`` is not deprecated.
diff --git a/setup.py b/setup.py
index de6cd3f0..e2a69912 100644
--- a/setup.py
+++ b/setup.py
@@ -51,7 +51,7 @@ if (
             # Use the stable ABI so our wheels are compatible across python
             # versions.
             py_limited_api=can_use_limited_api,
-            define_macros=[("Py_LIMITED_API", "0x03090000")],
+            define_macros=[("Py_LIMITED_API", "0x03090000")] if can_use_limited_api else [],
         )
     ]
 
diff --git a/tornado/__init__.py b/tornado/__init__.py
index 536f7c3f..d3ad3bea 100644
--- a/tornado/__init__.py
+++ b/tornado/__init__.py
@@ -22,8 +22,8 @@
 # is zero for an official release, positive for a development branch,
 # or negative for a release candidate or beta (after the base version
 # number has been incremented)
-version = "6.5.dev1"
-version_info = (6, 5, 0, -100)
+version = "6.5b1"
+version_info = (6, 5, 0, -99)
 
 import importlib
 import typing
diff --git a/tornado/gen.py b/tornado/gen.py
index d12f32fb..3dab733c 100644
--- a/tornado/gen.py
+++ b/tornado/gen.py
@@ -91,10 +91,22 @@ except ImportError:
     contextvars = None  # type: ignore
 
 import typing
-from typing import Union, Any, Callable, List, Type, Tuple, Awaitable, Dict, overload
+from typing import (
+    Mapping,
+    Union,
+    Any,
+    Callable,
+    List,
+    Type,
+    Tuple,
+    Awaitable,
+    Dict,
+    Sequence,
+    overload,
+)
 
 if typing.TYPE_CHECKING:
-    from typing import Sequence, Deque, Optional, Set, Iterable  # noqa: F401
+    from typing import Deque, Optional, Set, Iterable  # noqa: F401
 
 _T = typing.TypeVar("_T")
 
@@ -426,7 +438,7 @@ class WaitIterator:
 
 
 def multi(
-    children: Union[List[_Yieldable], Dict[Any, _Yieldable]],
+    children: Union[Sequence[_Yieldable], Mapping[Any, _Yieldable]],
     quiet_exceptions: "Union[Type[Exception], Tuple[Type[Exception], ...]]" = (),
 ) -> "Union[Future[List], Future[Dict]]":
     """Runs multiple asynchronous operations in parallel.
@@ -480,7 +492,7 @@ Multi = multi
 
 
 def multi_future(
-    children: Union[List[_Yieldable], Dict[Any, _Yieldable]],
+    children: Union[Sequence[_Yieldable], Mapping[Any, _Yieldable]],
     quiet_exceptions: "Union[Type[Exception], Tuple[Type[Exception], ...]]" = (),
 ) -> "Union[Future[List], Future[Dict]]":
     """Wait for multiple asynchronous futures in parallel.
diff --git a/tornado/httputil.py b/tornado/httputil.py
index 5d05cacb..7044aca0 100644
--- a/tornado/httputil.py
+++ b/tornado/httputil.py
@@ -33,7 +33,7 @@ import time
 import unicodedata
 from urllib.parse import urlencode, urlparse, urlunparse, parse_qsl
 
-from tornado.escape import native_str, parse_qs_bytes, utf8
+from tornado.escape import native_str, parse_qs_bytes, utf8, to_unicode
 from tornado.log import gen_log
 from tornado.util import ObjectDict, unicode_type
 
@@ -100,6 +100,12 @@ class _ABNF:
     # RFC 9110 (HTTP Semantics)
     obs_text = re.compile(r"[\x80-\xFF]")
     field_vchar = re.compile(rf"(?:{VCHAR.pattern}|{obs_text.pattern})")
+    # Not exactly from the RFC to simplify and combine field-content and field-value.
+    field_value = re.compile(
+        rf"|"
+        rf"{field_vchar.pattern}|"
+        rf"{field_vchar.pattern}(?:{field_vchar.pattern}| |\t)*{field_vchar.pattern}"
+    )
     tchar = re.compile(r"[!#$%&'*+\-.^_`|~0-9A-Za-z]")
     token = re.compile(rf"{tchar.pattern}+")
     field_name = token
@@ -195,6 +201,10 @@ class HTTPHeaders(StrMutableMapping):
         """Adds a new value for the given key."""
         if not _ABNF.field_name.fullmatch(name):
             raise HTTPInputError("Invalid header name %r" % name)
+        if not _ABNF.field_value.fullmatch(to_unicode(value)):
+            # TODO: the fact we still support bytes here (contrary to type annotations)
+            # and still test for it should probably be changed.
+            raise HTTPInputError("Invalid header value %r" % value)
         norm_name = _normalize_header(name)
         self._last_key = norm_name
         if norm_name in self:
@@ -254,6 +264,8 @@ class HTTPHeaders(StrMutableMapping):
             if self._last_key is None:
                 raise HTTPInputError("first header line cannot start with whitespace")
             new_part = " " + line.strip(HTTP_WHITESPACE)
+            if not _ABNF.field_value.fullmatch(new_part[1:]):
+                raise HTTPInputError("Invalid header continuation %r" % new_part)
             self._as_list[self._last_key][-1] += new_part
             self._dict[self._last_key] += new_part
         else:
diff --git a/tornado/routing.py b/tornado/routing.py
index ee81f977..245070c7 100644
--- a/tornado/routing.py
+++ b/tornado/routing.py
@@ -184,7 +184,18 @@ from tornado.escape import url_escape, url_unescape, utf8
 from tornado.log import app_log
 from tornado.util import basestring_type, import_object, re_unescape, unicode_type
 
-from typing import Any, Union, Optional, Awaitable, List, Dict, Pattern, Tuple, overload
+from typing import (
+    Any,
+    Union,
+    Optional,
+    Awaitable,
+    List,
+    Dict,
+    Pattern,
+    Tuple,
+    overload,
+    Sequence,
+)
 
 
 class Router(httputil.HTTPServerConnectionDelegate):
@@ -286,7 +297,7 @@ class _DefaultMessageDelegate(httputil.HTTPMessageDelegate):
 
 # _RuleList can either contain pre-constructed Rules or a sequence of
 # arguments to be passed to the Rule constructor.
-_RuleList = List[
+_RuleList = Sequence[
     Union[
         "Rule",
         List[Any],  # Can't do detailed typechecking of lists.
diff --git a/tornado/speedups.c b/tornado/speedups.c
index 525d6603..992c29c1 100644
--- a/tornado/speedups.c
+++ b/tornado/speedups.c
@@ -51,20 +51,36 @@ static PyObject* websocket_mask(PyObject* self, PyObject* args) {
     return result;
 }
 
+static int speedups_exec(PyObject *module) {
+    return 0;
+}
+
 static PyMethodDef methods[] = {
     {"websocket_mask",  websocket_mask, METH_VARARGS, ""},
     {NULL, NULL, 0, NULL}
 };
 
+static PyModuleDef_Slot slots[] = {
+    {Py_mod_exec, speedups_exec},
+#if (!defined(Py_LIMITED_API) && PY_VERSION_HEX >= 0x030c0000) || Py_LIMITED_API >= 0x030c0000
+    {Py_mod_multiple_interpreters, Py_MOD_PER_INTERPRETER_GIL_SUPPORTED},
+#endif
+#if (!defined(Py_LIMITED_API) && PY_VERSION_HEX >= 0x030d0000) || Py_LIMITED_API >= 0x030d0000
+    {Py_mod_gil, Py_MOD_GIL_NOT_USED},
+#endif
+    {0, NULL}
+};
+
 static struct PyModuleDef speedupsmodule = {
    PyModuleDef_HEAD_INIT,
    "speedups",
    NULL,
-   -1,
-   methods
+   0,
+   methods,
+   slots,
 };
 
 PyMODINIT_FUNC
 PyInit_speedups(void) {
-    return PyModule_Create(&speedupsmodule);
+    return PyModuleDef_Init(&speedupsmodule);
 }
diff --git a/tornado/web.py b/tornado/web.py
index 289c6738..0303f547 100644
--- a/tornado/web.py
+++ b/tornado/web.py
@@ -284,7 +284,8 @@ class RequestHandler:
         """Called at the beginning of a request before  `get`/`post`/etc.
 
         Override this method to perform common initialization regardless
-        of the request method.
+        of the request method. There is no guarantee that ``prepare`` will
+        be called if an error occurs that is handled by the framework.
 
         Asynchronous support: Use ``async def`` or decorate this method with
         `.gen.coroutine` to make it asynchronous.
@@ -299,10 +300,14 @@ class RequestHandler:
     def on_finish(self) -> None:
         """Called after the end of a request.
 
-        Override this method to perform cleanup, logging, etc.
-        This method is a counterpart to `prepare`.  ``on_finish`` may
-        not produce any output, as it is called after the response
-        has been sent to the client.
+        Override this method to perform cleanup, logging, etc. This method is primarily intended as
+        a counterpart to `prepare`. However, there are a few error cases where ``on_finish`` may be
+        called when ``prepare`` has not. (These are considered bugs and may be fixed in the future,
+        but for now you may need to check to see if the initialization work done in ``prepare`` has
+        occurred)
+
+        ``on_finish`` may not produce any output, as it is called after the response has been sent
+        to the client.
         """
         pass
 


Test patch:
diff --git a/tornado/test/httputil_test.py b/tornado/test/httputil_test.py
index 221c1dcb..30fbec4d 100644
--- a/tornado/test/httputil_test.py
+++ b/tornado/test/httputil_test.py
@@ -303,6 +303,17 @@ Foo: even
         data = "Foo: bar\r\n\fasdf"
         self.assertRaises(HTTPInputError, HTTPHeaders.parse, data)
 
+    def test_forbidden_ascii_characters(self):
+        # Control characters and ASCII whitespace other than space, tab, and CRLF are not allowed in
+        # headers.
+        for c in range(0xFF):
+            data = f"Foo: bar{chr(c)}baz\r\n"
+            if c == 0x09 or (c >= 0x20 and c != 0x7F):
+                headers = HTTPHeaders.parse(data)
+                self.assertEqual(headers["Foo"], f"bar{chr(c)}baz")
+            else:
+                self.assertRaises(HTTPInputError, HTTPHeaders.parse, data)
+
     def test_unicode_newlines(self):
         # Ensure that only \r\n is recognized as a header separator, and not
         # the other newline-like unicode characters.
@@ -311,10 +322,13 @@ Foo: even
         # and cpython's unicodeobject.c (which defines the implementation
         # of unicode_type.splitlines(), and uses a different list than TR13).
         newlines = [
-            "\u001b",  # VERTICAL TAB
-            "\u001c",  # FILE SEPARATOR
-            "\u001d",  # GROUP SEPARATOR
-            "\u001e",  # RECORD SEPARATOR
+            # The following ascii characters are sometimes treated as newline-like,
+            # but they're disallowed in HTTP headers. This test covers unicode
+            # characters that are permitted in headers (under the obs-text rule).
+            # "\u001b",  # VERTICAL TAB
+            # "\u001c",  # FILE SEPARATOR
+            # "\u001d",  # GROUP SEPARATOR
+            # "\u001e",  # RECORD SEPARATOR
             "\u0085",  # NEXT LINE
             "\u2028",  # LINE SEPARATOR
             "\u2029",  # PARAGRAPH SEPARATOR
@@ -363,13 +377,16 @@ Foo: even
             self.assertEqual(expected, list(headers.get_all()))
 
     def test_optional_cr(self):
+        # Bare CR is  not a valid line separator
+        with self.assertRaises(HTTPInputError):
+            HTTPHeaders.parse("CRLF: crlf\r\nLF: lf\nCR: cr\rMore: more\r\n")
+
         # Both CRLF and LF should be accepted as separators. CR should not be
-        # part of the data when followed by LF, but it is a normal char
-        # otherwise (or should bare CR be an error?)
-        headers = HTTPHeaders.parse("CRLF: crlf\r\nLF: lf\nCR: cr\rMore: more\r\n")
+        # part of the data when followed by LF.
+        headers = HTTPHeaders.parse("CRLF: crlf\r\nLF: lf\nMore: more\r\n")
         self.assertEqual(
             sorted(headers.get_all()),
-            [("Cr", "cr\rMore: more"), ("Crlf", "crlf"), ("Lf", "lf")],
+            [("Crlf", "crlf"), ("Lf", "lf"), ("More", "more")],
         )
 
     def test_copy(self):