Repository: twisted/twisted

Issue:
DeprecationWarning: "'cgi' is deprecated and slated for removal in Python 3.13"
**Describe the incorrect behavior you saw**
The currently used package "cgi" will be removed in Python 3.13.

**Describe how to cause this behavior**
```
> python -W always
Python 3.11.0 (main, Feb  3 2023, 12:28:31) [MSC v.1929 64 bit (AMD64)] on win32
>>> from twisted.web import http
...\site-packages\twisted\web\http.py:103: DeprecationWarning: 'cgi' is deprecated and slated for removal in Python 3.13
  import cgi
```

**Describe the correct behavior you'd like to see**
No warning.

**Testing environment**
 - Windows: 11
 - Python: 3.11.0
 - Twisted: 22.10.0



Discussion:
Thanks for the report. I am not sure what is the bug here :)

I see that you are using 3.11 .
I guess that the `cgi` module is still present and Twisted works as expected.

I think that this is more of a refactoring task / future-proof task, rather than a bug.

Is there something broken with Twisted?

Happy to review a PR in which the `cgi` module is removed and replaced with something else.
Yes sorry, it's not a bug. Everything's working fine! I just didn't know how to mark it as a refactoring task afterwards.
Hello,
we've  started building Fedora packages with Python 3.13.0 alphas and there's quite a lot failures when building twisted. 

```
FAILED (skips=640, failures=14, errors=51, successes=9085)
```

They're not only related to the removed `cgi`, see full build logs: https://download.copr.fedorainfracloud.org/results/@python/python3.13/fedora-rawhide-x86_64/06770350-python-twisted/builder-live.log.gz

Many thanks Karolina for the report.

The `cgi` issue should already be fixed in `trunk` ... but not yet released 
This was done in https://github.com/twisted/twisted/issues/12015

Also, the skip test `unittest` part was already fixed in `trunk` as part of #12054

-------

Since `cgi` is fixed , I am going to close this and we can continue reporting python 3.13 errors over #11750 and try to fix the issues in separate/dedicated tickets.
> Since `cgi` is fixed , I am going to close this...

I see a cgi import in:

https://github.com/twisted/twisted/blob/c0c6440827c47797415b6ddf5bbf0b60a203da02/src/twisted/web/http.py#L104

This is in trunk. 
True. Many thanks.

It is used here https://github.com/twisted/twisted/blob/c0c6440827c47797415b6ddf5bbf0b60a203da02/src/twisted/web/http.py#L991

We should fix it.

Resolving patch:
diff --git a/src/twisted/web/http.py b/src/twisted/web/http.py
--- a/src/twisted/web/http.py
+++ b/src/twisted/web/http.py
@@ -101,13 +101,13 @@
 import base64
 import binascii
 import calendar
-import cgi
 import math
 import os
 import re
 import tempfile
 import time
 import warnings
+from email import message_from_bytes
 from email.message import EmailMessage
 from io import BytesIO
 from typing import AnyStr, Callable, List, Optional, Tuple
@@ -226,18 +226,40 @@
 monthname_lower = [name and name.lower() for name in monthname]
 
 
-def _parseContentType(line: bytes) -> tuple[bytes, dict[str, bytes]]:
+def _parseContentType(line: bytes) -> bytes:
+    """
+    Parse the Content-Type header.
+    """
     msg = EmailMessage()
     msg["content-type"] = line.decode("charmap")
-
     key = msg.get_content_type()
-    pdict = msg["content-type"].params
-
-    # We want the key as bytes, and cgi.parse_multipart (which consumes
-    # pdict) expects a dict of str keys but bytes values
     encodedKey = key.encode("charmap")
-    pdict = {x: y.encode("charmap") for x, y in pdict.items()}
-    return (encodedKey, pdict)
+    return encodedKey
+
+
+class _MultiPartParseException(Exception):
+    """
+    Failed to parse the multipart/form-data payload.
+    """
+
+
+def _getMultiPartArgs(content: bytes, ctype: bytes) -> dict[bytes, list[bytes]]:
+    """
+    Parse the content of a multipart/form-data request.
+    """
+    result = {}
+    multiPartHeaders = b"MIME-Version: 1.0\r\n" + b"Content-Type: " + ctype + b"\r\n"
+    msg = message_from_bytes(multiPartHeaders + content)
+    if not msg.is_multipart():
+        raise _MultiPartParseException("Not a multipart.")
+
+    for part in msg.get_payload():
+        name = part.get_param("name", header="content-disposition")
+        if not name:
+            continue
+        payload = part.get_payload(decode=True)
+        result[name.encode("utf8")] = [payload]
+    return result
 
 
 def urlparse(url):
@@ -272,6 +294,9 @@ def parse_qs(qs, keep_blank_values=0, strict_parsing=0):
     """
     Like C{cgi.parse_qs}, but with support for parsing byte strings on Python 3.
 
+    This was created to help with Python 2 to Python 3 migration.
+    Consider using L{urllib.parse.parse_qs}.
+
     @type qs: C{bytes}
     """
     d = {}
@@ -978,47 +1003,18 @@ def requestReceived(self, command, path, version):
 
         if self.method == b"POST" and ctype and clength:
             mfd = b"multipart/form-data"
-            key, pdict = _parseContentType(ctype)
-            # This weird CONTENT-LENGTH param is required by
-            # cgi.parse_multipart() in some versions of Python 3.7+, see
-            # bpo-29979. It looks like this will be relaxed and backported, see
-            # https://github.com/python/cpython/pull/8530.
-            pdict["CONTENT-LENGTH"] = clength
+            key = _parseContentType(ctype)
             if key == b"application/x-www-form-urlencoded":
                 args.update(parse_qs(self.content.read(), 1))
             elif key == mfd:
                 try:
-                    cgiArgs = cgi.parse_multipart(
-                        self.content,
-                        pdict,
-                        encoding="utf8",
-                        errors="surrogateescape",
-                    )
-
-                    # The parse_multipart function on Python 3.7+
-                    # decodes the header bytes as iso-8859-1 and
-                    # decodes the body bytes as utf8 with
-                    # surrogateescape -- we want bytes
-                    self.args.update(
-                        {
-                            x.encode("iso-8859-1"): [
-                                z.encode("utf8", "surrogateescape")
-                                if isinstance(z, str)
-                                else z
-                                for z in y
-                            ]
-                            for x, y in cgiArgs.items()
-                            if isinstance(x, str)
-                        }
-                    )
-                except Exception as e:
-                    # It was a bad request, or we got a signal.
+                    self.content.seek(0)
+                    content = self.content.read()
+                    self.args.update(_getMultiPartArgs(content, ctype))
+                except _MultiPartParseException:
+                    # It was a bad request.
                     self.channel._respondToBadRequestAndDisconnect()
-                    if isinstance(e, (TypeError, ValueError, KeyError)):
-                        return
-                    else:
-                        # If it's not a userspace error from CGI, reraise
-                        raise
+                    return
 
             self.content.seek(0, 0)
 


Test patch: