CVE-2023-41419 (Python, published 2023-09-25T12:15Z)
Description: value: An issue in Gevent before version 23.9.0 allows a remote attacker to escalate privileges via a crafted script to the WSGIServer component.
Severity: nan (CVSS v3 base score: 9.8)
Weakness: NVD-CWE-noinfo \u2014 Insufficient Information
There is insufficient information about the issue to classify it; details are unkown or unspecified.
Repository: https://github.com/gevent/gevent
Files changed:
docs/changes/1989.bugfix
src/gevent/pywsgi.py
src/gevent/subprocess.py
src/gevent/testing/testcase.py
src/gevent/tests/test__pywsgi.py
Commit message: gevent.pywsgi: Much improved handling of chunk trailers.

Validation is much stricter to the specification.

Fixes #1989


Vulnerable code:
   and is not intended for direct exposure to the public Internet.
try:
    from urllib import unquote
except ImportError:
    from urllib.parse import unquote # python 2 pylint:disable=import-error,no-name-in-module
_WEEKDAYNAME = ["Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun"]
_MONTHNAME = [None,  # Dummy so we can use 1-based month numbers
              "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"]
_INTERNAL_ERROR_HEADERS = [('Content-Type', 'text/plain'),
                           ('Connection', 'close'),
                           ('Content-Length', str(len(_INTERNAL_ERROR_BODY)))]
_BAD_REQUEST_HEADERS = [('Content-Type', 'text/plain'),
                        ('Connection', 'close'),
                        ('Content-Length', str(len(_BAD_REQUEST_BODY)))]
        # Here's the production for a chunk:
        # (http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html)
        #   chunk          = chunk-size [ chunk-extension ] CRLF
        #                    chunk-data CRLF
        #   chunk-size     = 1*HEX
        #   chunk-extension= *( ";" chunk-ext-name [ "=" chunk-ext-val ] )
        #   chunk-ext-name = token
        #   chunk-ext-val  = token | quoted-string

        # To cope with malicious or broken clients that fail to send valid
        # chunk lines, the strategy is to read character by character until we either reach
        # a ; or newline. If at any time we read a non-HEX digit, we bail. If we hit a
        # ;, indicating an chunk-extension, we'll read up to the next
        # MAX_REQUEST_LINE characters
        # looking for the CRLF, and if we don't find it, we bail. If we read more than 16 hex characters,
        # (the number needed to represent a 64-bit chunk size), we bail (this protects us from
        # a client that sends an infinite stream of `F`, for example).
            if char == b'\r':
                break
            if char == b';':
            if char not in _HEX:
            if buf.tell() > 16:
            if char != b'\n':
                raise _InvalidClientInput("Line didn't end in CRLF")
            return int(buf.getvalue(), 16)
                    rfile.readline()
                if self.chunk_length == 0:
                    # Last chunk. Terminates with a CRLF.
                    rfile.readline()
                raise _InvalidClientRequest('Expected GET method: %r' % (raw_requestline,))
                    # Don't let exceptions during discarding
        except _InvalidClientInput:
        if not isinstance(ex, ValueError):
            self.log_error(ex.formatted_message)
            self.log_error('Invalid request: %s', str(ex) or ex.__class__.__name__)
        >>> print(check_output(["/bin/sh", "-c",
        ...              stderr=STDOUT).decode('ascii').strip())
        ls: non_existent_file: No such file or directory
                value = _wrap_timeout(timeout, value)
try:
    from urllib.parse import parse_qs
except ImportError:
    # Python 2
    from urlparse import parse_qs
try:
    # On Python 2, we want the C-optimized version if
    # available; it has different corner-case behaviour than
    # the Python implementation, and it used by socket.makefile
    # by default.
    from cStringIO import StringIO
except ImportError:
    from io import BytesIO as StringIO

Fixed code:
Make ``gevent.pywsgi`` comply more closely with the HTTP specification
for chunked transfer encoding. In particular, we are much stricter
about trailers, and trailers that are invalid (too long or featuring
disallowed characters) forcibly close the connection to the client
*after* the results have been sent.

Trailers otherwise continue to be ignored and are not available to the
WSGI application.

Previously, carefully crafted invalid trailers in chunked requests on
keep-alive connections might appear as two requests to
``gevent.pywsgi``. Because this was handled exactly as a normal
keep-alive connection with two requests, the WSGI application should
handle it normally. However, if you were counting on some upstream
server to filter incoming requests based on paths or header fields,
and the upstream server simply passed trailers through without
validating them, then this embedded second request would bypass those
checks. (If the upstream server validated that the trailers meet the
HTTP specification, this could not occur, because characters that are
required in an HTTP request, like a space, are not allowed in
trailers.) CVE-2023-41419 was reserved for this.

Our thanks to the original reporters, Keran Mu
(mkr22@mails.tsinghua.edu.cn) and Jianjun Chen
(jianjun@tsinghua.edu.cn), from Tsinghua University and Zhongguancun
Laboratory.
   and is not intended for direct exposure to the public Internet. For production
   usage on the Internet, either choose a production-strength server such as
   gunicorn, or put a reverse proxy between gevent and the Internet.

.. versionchanged:: NEXT

   Complies more closely with the HTTP specification for chunked transfer encoding.
   In particular, we are much stricter about trailers, and trailers that
   are invalid (too long or featuring disallowed characters) forcibly close
   the connection to the client *after* the results have been sent.

   Trailers otherwise continue to be ignored and are not available to the
   WSGI application.
from urllib.parse import unquote
_WEEKDAYNAME = ("Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun")
_MONTHNAME = (None,  # Dummy so we can use 1-based month numbers
              "Jul", "Aug", "Sep", "Oct", "Nov", "Dec")
# The characters allowed in "token" rules.

# token          = 1*tchar
# tchar          = "!" / "#" / "$" / "%" / "&" / "'" / "*"
#                / "+" / "-" / "." / "^" / "_" / "`" / "|" / "~"
#                / DIGIT / ALPHA
#                ; any VCHAR, except delimiters
# ALPHA          =  %x41-5A / %x61-7A   ; A-Z / a-z
_ALLOWED_TOKEN_CHARS = frozenset(
    # Remember we have to be careful because bytestrings
    # inexplicably iterate as integers, which are not equal to bytes.

    # explicit chars then DIGIT
    (c.encode('ascii') for c in "!#$%&'*+-.^_`|~0123456789")
    # Then we add ALPHA
) | {c.encode('ascii') for c in string.ascii_letters}
assert b'A' in _ALLOWED_TOKEN_CHARS


_INTERNAL_ERROR_HEADERS = (
    ('Content-Type', 'text/plain'),
    ('Connection', 'close'),
    ('Content-Length', str(len(_INTERNAL_ERROR_BODY)))
)
_BAD_REQUEST_HEADERS = (
    ('Content-Type', 'text/plain'),
    ('Connection', 'close'),
    ('Content-Length', str(len(_BAD_REQUEST_BODY)))
)
        # Here's the production for a chunk (actually the whole body):
        # (https://www.rfc-editor.org/rfc/rfc7230#section-4.1)

        # chunked-body   = *chunk
        #                  last-chunk
        #                  trailer-part
        #                  CRLF
        #
        # chunk          = chunk-size [ chunk-ext ] CRLF
        #                  chunk-data CRLF
        # chunk-size     = 1*HEXDIG
        # last-chunk     = 1*("0") [ chunk-ext ] CRLF
        # trailer-part   = *( header-field CRLF )
        # chunk-data     = 1*OCTET ; a sequence of chunk-size octets

        # To cope with malicious or broken clients that fail to send
        # valid chunk lines, the strategy is to read character by
        # character until we either reach a ; or newline. If at any
        # time we read a non-HEX digit, we bail. If we hit a ;,
        # indicating an chunk-extension, we'll read up to the next
        # MAX_REQUEST_LINE characters ("A server ought to limit the
        # total length of chunk extensions received") looking for the
        # CRLF, and if we don't find it, we bail. If we read more than
        # 16 hex characters, (the number needed to represent a 64-bit
        # chunk size), we bail (this protects us from a client that
        # sends an infinite stream of `F`, for example).

            if char in (
                b'\r', # Beginning EOL
                b';', # Beginning extension
            ):
            if char not in _HEX: # Invalid data.


            if buf.tell() > 16: # Too many hex bytes
            self.__read_chunk_size_crlf(rfile, newline_only=True)
            result = int(buf.getvalue(), 16)
            if result == 0:
                # The only time a chunk size of zero is allowed is the final
                # chunk. It is either followed by another \r\n, or some trailers
                # which are then followed by \r\n.
                while self.__read_chunk_trailer(rfile):
                    pass
            return result

    # Trailers have the following production (they are a header-field followed by CRLF)
    # See above for the definition of "token".
    #
    # header-field   = field-name ":" OWS field-value OWS
    # field-name     = token
    # field-value    = *( field-content / obs-fold )
    # field-content  = field-vchar [ 1*( SP / HTAB ) field-vchar ]
    # field-vchar    = VCHAR / obs-text
    # obs-fold       = CRLF 1*( SP / HTAB )
    #                ; obsolete line folding
    #                ; see Section 3.2.4


    def __read_chunk_trailer(self, rfile, ):
        # With rfile positioned just after a \r\n, read a trailer line.
        # Return a true value if a non-empty trailer was read, and
        # return false if an empty trailer was read (meaning the trailers are
        # done).
        # If a single line exceeds the MAX_REQUEST_LINE, raise an exception.
        # If the field-name portion contains invalid characters, raise an exception.

        i = 0
        empty = True
        seen_field_name = False
        while i < MAX_REQUEST_LINE:
            if char == b'\r':
                # Either read the next \n or raise an error.
                self.__read_chunk_size_crlf(rfile, newline_only=True)
                break
            # Not a \r, so we are NOT an empty chunk.
            empty = False
            if char == b':' and i > 0:
                # We're ending the field-name part; stop validating characters.
                # Unless : was the first character...
                seen_field_name = True
            if not seen_field_name and char not in _ALLOWED_TOKEN_CHARS:
                raise _InvalidClientInput('Invalid token character: %r' % (char,))
            i += 1
        else:
            # We read too much
            self._chunked_input_error = True
            raise _InvalidClientInput("Too large chunk trailer")
        return not empty

    def __read_chunk_size_crlf(self, rfile, newline_only=False):
        # Also for safety, correctly verify that we get \r\n when expected.
        if not newline_only:
            char = rfile.read(1)
            if char != b'\r':
                raise _InvalidClientInput("Line didn't end in CRLF: %r" % (char,))
        char = rfile.read(1)
        if char != b'\n':
            self._chunked_input_error = True
            raise _InvalidClientInput("Line didn't end in LF: %r" % (char,))
                    self.__read_chunk_size_crlf(rfile)
                # If chunk_length was 0, we already read any trailers and
                # validated that we have ended with \r\n\r\n.

                raise _InvalidClientRequest('Expected GET method; Got command=%r; path=%r; raw=%r' % (
                    self.command, self.path, raw_requestline,))
                except _InvalidClientInput:
                    # This one is deliberately raised to the outer
                    # scope, because, with the incoming stream in some bad state,
                    # we can't be sure we can synchronize and properly parse the next
                    # request.
                    raise
                    # Don't let socket exceptions during discarding
        except _InvalidClientInput as ex:
            # DO log this one because:
            # - Some of the data may have been read and acted on by the
            #   application;
            # - The response may or may not have been sent;
            # - It's likely that the client is bad, or malicious, and
            #   users might wish to take steps to block the client.
            self._handle_client_error(ex)
            self.close_connection = True
        if not isinstance(ex, (ValueError, _InvalidClientInput)):
            # _InvalidClientRequest is a ValueError; _InvalidClientInput is an IOError.
            # However, the error messages do not include the requesting IP
            # necessarily, so we do add that.
            self.log_error('(from %s) %s', self.client_address, ex.formatted_message)
            self.log_error('Invalid request (from %s): %s',
                           self.client_address,
                           str(ex) or ex.__class__.__name__)
        >>> output = check_output(["/bin/sh", "-c",
        ...              stderr=STDOUT).decode('ascii').strip()
        >>> print(output.rsplit(':', 1)[1].strip())
        No such file or directory
                #value = _wrap_timeout(timeout, value)
from urllib.parse import parse_qs
from io import BytesIO as StringIO

        """
        Read an HTTP response, optionally perform assertions,
        and return the Response object.
        """
    calls = 0

    def setUp(self):
        super().setUp()
        self.calls = 0

        self.calls += 1
        return [b'We should not get here', env['PATH_INFO'].encode('ascii')]

    def test_trailers_keepalive_ignored(self):
        # Trailers after a chunk are ignored.
        data = (
            b'POST /a HTTP/1.1\r\n'
            b'Host: localhost\r\n'
            b'Connection: keep-alive\r\n'
            b'Transfer-Encoding: chunked\r\n'
            b'\r\n'
            b'2\r\noh\r\n'
            b'4\r\n hai\r\n'
            b'0\r\n' # last-chunk
            # Normally the final CRLF would go here, but if you put in a
            # trailer, it doesn't.
            b'trailer1: value1\r\n'
            b'trailer2: value2\r\n'
            b'\r\n' # Really terminate the chunk.
            b'POST /a HTTP/1.1\r\n'
            b'Host: localhost\r\n'
            b'Connection: close\r\n'
            b'Transfer-Encoding: chunked\r\n'
            b'\r\n'
            b'2\r\noh\r\n'
            b'4\r\n bye\r\n'
            b'0\r\n' # last-chunk
        )
        with self.makefile() as fd:
            fd.write(data)
            read_http(fd, body='oh hai')
            read_http(fd, body='oh bye')

        self.assertEqual(self.calls, 2)

    def test_trailers_too_long(self):
        # Trailers after a chunk are ignored.
        data = (
            b'POST /a HTTP/1.1\r\n'
            b'Host: localhost\r\n'
            b'Connection: keep-alive\r\n'
            b'Transfer-Encoding: chunked\r\n'
            b'\r\n'
            b'2\r\noh\r\n'
            b'4\r\n hai\r\n'
            b'0\r\n' # last-chunk
            # Normally the final CRLF would go here, but if you put in a
            # trailer, it doesn't.
            b'trailer2: value2' # not lack of \r\n
        )
        data += b't' * pywsgi.MAX_REQUEST_LINE
        # No termination, because we detect the trailer as being too
        # long and abort the connection.
        with self.makefile() as fd:
            fd.write(data)
            read_http(fd, body='oh hai')
            with self.assertRaises(ConnectionClosed):
                read_http(fd, body='oh bye')

    def test_trailers_request_smuggling_missing_last_chunk_keep_alive(self):
        # When something that looks like a request line comes in the trailer
        # as the first line, immediately after an invalid last chunk.
        # We detect this and abort the connection, because the
        # whitespace in the GET line isn't a legal part of a trailer.
        # If we didn't abort the connection, then, because we specified
        # keep-alive, the server would be hanging around waiting for more input.
        data = (
            b'POST /a HTTP/1.1\r\n'
            b'Host: localhost\r\n'
            b'Connection: keep-alive\r\n'
            b'Transfer-Encoding: chunked\r\n'
            b'\r\n'
            b'2\r\noh\r\n'
            b'4\r\n hai\r\n'
            b'0' # last-chunk, but missing the \r\n
            # Normally the final CRLF would go here, but if you put in a
            # trailer, it doesn't.
            # b'\r\n'
            b'GET /path2?a=:123 HTTP/1.1\r\n'
            b'Host: a.com\r\n'
            b'Connection: close\r\n'
            b'\r\n'
        )
        with self.makefile() as fd:
            fd.write(data)
            read_http(fd, body='oh hai')
            with self.assertRaises(ConnectionClosed):
                read_http(fd)

        self.assertEqual(self.calls, 1)

    def test_trailers_request_smuggling_missing_last_chunk_close(self):
        # Same as the above, except the trailers are actually valid
        # and since we ask to close the connection we don't get stuck
        # waiting for more input.
        data = (
            b'POST /a HTTP/1.1\r\n'
            b'Host: localhost\r\n'
            b'Connection: close\r\n'
            b'Transfer-Encoding: chunked\r\n'
            b'\r\n'
            b'2\r\noh\r\n'
            b'4\r\n hai\r\n'
            b'0\r\n' # last-chunk
            # Normally the final CRLF would go here, but if you put in a
            # trailer, it doesn't.
            # b'\r\n'
            b'GETpath2a:123 HTTP/1.1\r\n'
            b'Host: a.com\r\n'
            b'Connection: close\r\n'
            b'\r\n'
        )
        with self.makefile() as fd:
            fd.write(data)
            read_http(fd, body='oh hai')
            with self.assertRaises(ConnectionClosed):
                read_http(fd)

    def test_trailers_request_smuggling_header_first(self):
        # When something that looks like a header comes in the first line.
        data = (
            b'POST /a HTTP/1.1\r\n'
            b'Host: localhost\r\n'
            b'Connection: keep-alive\r\n'
            b'Transfer-Encoding: chunked\r\n'
            b'\r\n'
            b'2\r\noh\r\n'
            b'4\r\n hai\r\n'
            b'0\r\n' # last-chunk, but only one CRLF
            b'Header: value\r\n'
            b'GET /path2?a=:123 HTTP/1.1\r\n'
            b'Host: a.com\r\n'
            b'Connection: close\r\n'
            b'\r\n'
        )
        with self.makefile() as fd:
            fd.write(data)
            read_http(fd, body='oh hai')
            with self.assertRaises(ConnectionClosed):
                read_http(fd, code=400)

        self.assertEqual(self.calls, 1)

    def test_trailers_request_smuggling_request_terminates_then_header(self):
        data = (
            b'POST /a HTTP/1.1\r\n'
            b'Host: localhost\r\n'
            b'Connection: keep-alive\r\n'
            b'Transfer-Encoding: chunked\r\n'
            b'\r\n'
            b'2\r\noh\r\n'
            b'4\r\n hai\r\n'
            b'0\r\n' # last-chunk
            b'\r\n'
            b'Header: value'
            b'GET /path2?a=:123 HTTP/1.1\r\n'
            b'Host: a.com\r\n'
            b'Connection: close\r\n'
            b'\r\n'
        )
        with self.makefile() as fd:
            fd.write(data)
            read_http(fd, body='oh hai')
            read_http(fd, code=400)

        self.assertEqual(self.calls, 1)