Repository: urllib3/urllib3

Issue:
Bug: Content-Disposition Header Lacking "filename" When "filename*" Is Present
#### Description:
When sending files to a server that does not understand "filename*" parameters inside the Content-Disposition header, the server will fail to find the file name because the "filename" parameter is not also included.

#### Justification:
According to [RFC 6266](https://tools.ietf.org/html/rfc6266#appendix-D), ideal header generators will include both a "filename" and a "filename*" parameter whenever the "filename*" form is required:
> – Include a "filename*" parameter where the desired filename cannot be expressed faithfully using the "filename" form.  Note that legacy user agents will not process this, and will fall back to using the "filename" parameter's content.
>
> – When a "filename*" parameter is sent, to also generate a "filename" parameter as a fallback for user agents that do not support the "filename*" form, if possible.  This can be done by substituting characters with US-ASCII sequences (e.g., Unicode character point U+00E4 (LATIN SMALL LETTER A WITH DIARESIS) by "ae").  Note that this may not be possible in some locales.
>
> – When a "filename" parameter is included as a fallback (as per above), "filename" should occur first, due to parsing problems in some existing implementations.

#### Why I Care:
This inconsistency caused a many-hour debugging session when trying to discover why file uploads to Google AppEngine blobstore stopped working when the requests library was upgraded (which bundles urllib3 with install). The eventual resolution was to urlencode the file name to a `str` before upload so that the "filename*" parameter was not added.

---

Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/39703440-bug-content-disposition-header-lacking-filename-when-filename-is-present?utm_campaign=plugin&utm_content=tracker%2F192525&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F192525&utm_medium=issues&utm_source=github).



Discussion:
So, we could definitely do this: we could aim to do so by encoding using an 'ignore' or 'replace' mode on the encoder. Alternatively, we could punycode.

I don't consider this enormously high priority because servers really *should* tolerate the `*` form. That means that I don't think this is going to bump any of my other work down, so anyone else who wants to is welcome to work on this.
Please note that `cgi.FieldStorage` doesn't support `filename*` form (in Python 2 at least), so all servers using the standard library's `cgi` module are actually broken. It's a bit annoying!
@sprat Pull requests to rectify this would be happily reviewed and accepted. Complaining gets us nowhere. Feel free to send a pull request to fix this.
I'm working on this.
Started doing some research. Created a Werkzeug application to echo the submitted `Content-Disposition` header.

```python
import werkzeug

form_html = """\
<!doctype html>
<meta charset=utf-8>
<form method=post enctype=multipart/form-data>
<input type=file multiple name=file>
<input type=submit>
</form>
"""


@werkzeug.Request.application
def app(request: werkzeug.Request):
    if request.method == "POST":
        for f in request.files.getlist("file"):
            print(repr(f.filename))

        return werkzeug.Response(status=204)

    return werkzeug.Response(form_html, content_type="text/html")


werkzeug.run_simple("localhost", 5000, app)
```

When a file named "basic.txt" is submitted from Firefox or Chromium, it prints:

```
form-data; name="file"; filename="basic.txt"
```

When a file named "ski ⛷.txt" is submitted from Firefox or Chromium, it prints:

```
form-data; name="file"; filename="ski ⛷.txt"
```

Browsers are providing one filename, and do not escape non-ASCII characters.
Using urllib3 to upload the file, the name is not encoded.

```python
import urllib3

http = urllib3.PoolManager()
http.request(
    "POST",
    "http://localhost:5000/",
    fields={"file": ("ski ⛷.txt", b"value")},
)
```

```
form-data; name="file"; filename="ski ⛷.txt"
```
[WHATWG HTML Standard, section 4.10.21.8, Multipart form data:](https://html.spec.whatwg.org/multipage/form-control-infrastructure.html#multipart-form-data)

> Return the byte sequence resulting from encoding the *entry list* using the rules described by RFC 7578, *Returning Values from Forms: `multipart/form-data`*

[RFC 7578, section 4.2, Content-Disposition Header Field for Each Part:](https://datatracker.ietf.org/doc/html/rfc7578#section-4.2)

> In most multipart types, the MIME header fields in each part are restricted to US-ASCII; for compatibility with those systems, file names normally visible to users MAY be encoded using the percent-encoding method in [Section 2](https://datatracker.ietf.org/doc/html/rfc7578#section-2), following how a "file:" URI \[[URI-SCHEME](https://datatracker.ietf.org/doc/html/rfc7578#ref-URI-SCHEME)\] might be encoded.
> 
> NOTE: The encoding method described in \[[RFC5987](https://datatracker.ietf.org/doc/html/rfc5987)\], which would add a "filename*" parameter to the Content-Disposition header field, MUST NOT be used.

The WHATWG HTML Standard says to follow RFC 7578 to encode `multipart/form-data`. The RFC states that *only* a `filename` key should be present, a `filename*` key is invalid. Therefore the suggestion in the original issue to send both keys is incorrect, `filename*` is only used when receiving files in a response from the server (Werkzeug's `send_file` supports this, for example).

The RFC suggests using UTF-8 percent encoding. However, browser forms are using HTML character references. I'm not clear if that's a limitation of browsers, maybe since it's HTML the filename is already HTML encoded by the time it's submitted, so it's already ASCII and doesn't get percent encoded.
Turns out urllib3 is already encoding values using `fields.format_header_param_html5`, which uses percent encoding, but only on control characters 0x00 - 0x1F, not arbitrary characters. It references the same multipart-form data standard above, but oddly, it says it "matches the behavior of curl and modern browsers". However, curl seems to have the same non-encoding behavior, and browsers don't use percent encoding, so this statement isn't correct.

```
curl -F "file=@ski ⛷.txt" http://localhost:5000
```

```
form-data; name="file"; filename="ski ⛷.txt"
```

This issue was opened before #1492, the old RFC 2231 encoder *does* incorrectly generate a `filename*` key. This issue was valid at the time it was reported, was invalidated by #1492, but still revealed other changes and deprecations that needed to happen.
Did some more investigation and discussion on Discord.

* After adding `<meta charset=utf-8>` to the test HTML, the browser began passing through the name unchanged like curl did. Seems the a legacy behavior when no encoding is set is to using char ref replacement.
* Neither browsers nor curl do anything special with control characters, all bytes are passed through unchanged.
* urllib3 percent encodes control characters, but no other characters
* RFC 7578 says "MAY be encoded", not "MUST" be encoded

Therefore, we've concluded that current behavior of browsers/curl/urllib3 is correct, the utf-8 filename can be passed through unchanged. Even though urllib3 *could* remove the percent encoding of control characters, it will leave it for now.
Did more digging. #1492 https://github.com/urllib3/urllib3/pull/1492/commits/131d0d5b14aede8d061e916e765866dee9eef9e5 added the control character quoting behavior, after @sethmlarson https://github.com/urllib3/urllib3/pull/1492#discussion_r264646123 noted that the WHATWG standard said to. This was a bit confusing, since the standard no longer mentions that. I tracked that down to https://github.com/whatwg/html/pull/6282, which removed mention of escaping anything except `"` and `\` since it didn't match what clients were actually doing.
I think since we're at a major release point, we should stop escaping control characters to match the current standard. I can make that change, or I can change the docs only. Note that users can always provide a more constrained file name, and that servers should always be treating the given name as untrusted and do processing on it regardless of what the client sends.

Resolving patch:
diff --git a/noxfile.py b/noxfile.py
--- a/noxfile.py
+++ b/noxfile.py
@@ -135,7 +135,7 @@ def lint(session):
 @nox.session()
 def mypy(session):
     """Run mypy."""
-    session.install("mypy")
+    session.install("mypy==0.812")
     session.run("mypy", "--version")
 
     session.log("mypy --strict src/urllib3")
diff --git a/src/urllib3/fields.py b/src/urllib3/fields.py
--- a/src/urllib3/fields.py
+++ b/src/urllib3/fields.py
@@ -1,12 +1,10 @@
 import email.utils
 import mimetypes
-import re
 from typing import (
     Callable,
     Dict,
     Iterable,
     Mapping,
-    Match,
     Optional,
     Sequence,
     Tuple,
@@ -49,7 +47,21 @@ def format_header_param_rfc2231(name: str, value: Union[str, bytes]) -> str:
         The value of the parameter, provided as ``bytes`` or `str``.
     :returns:
         An RFC-2231-formatted unicode string.
+
+    .. deprecated:: 2.0.0
+        Will be removed in urllib3 v3.0.0. This is not valid for
+        ``multipart/form-data`` header parameters.
     """
+    import warnings
+
+    warnings.warn(
+        "'format_header_param_rfc2231' is deprecated and will be "
+        "removed in urllib3 v3.0.0. This is not valid for "
+        "multipart/form-data header parameters.",
+        DeprecationWarning,
+        stacklevel=2,
+    )
+
     if isinstance(value, bytes):
         value = value.decode("utf-8")
 
@@ -68,60 +80,78 @@ def format_header_param_rfc2231(name: str, value: Union[str, bytes]) -> str:
     return value
 
 
-_HTML5_REPLACEMENTS = {
-    "\u0022": "%22",
-    # Replace "\" with "\\".
-    "\u005C": "\u005C\u005C",
-}
-
-# All control characters from 0x00 to 0x1F *except* 0x1B.
-_HTML5_REPLACEMENTS.update(
-    {chr(cc): f"%{cc:02X}" for cc in range(0x00, 0x1F + 1) if cc not in (0x1B,)}
-)
-
-
-def _replace_multiple(value: str, needles_and_replacements: Mapping[str, str]) -> str:
-    def replacer(match: Match[str]) -> str:
-        return needles_and_replacements[match.group(0)]
-
-    pattern = re.compile(
-        r"|".join([re.escape(needle) for needle in needles_and_replacements.keys()])
-    )
-
-    result = pattern.sub(replacer, value)
-
-    return result
-
-
-def format_header_param_html5(name: str, value: _TYPE_FIELD_VALUE) -> str:
+def format_multipart_header_param(name: str, value: _TYPE_FIELD_VALUE) -> str:
     """
-    Helper function to format and quote a single header parameter using the
-    HTML5 strategy.
+    Format and quote a single multipart header parameter.
 
-    Particularly useful for header parameters which might contain
-    non-ASCII values, like file names. This follows the `HTML5 Working Draft
-    Section 4.10.22.7`_ and matches the behavior of curl and modern browsers.
+    This follows the `WHATWG HTML Standard`_ as of 2021/06/10, matching
+    the behavior of current browser and curl versions. Values are
+    assumed to be UTF-8. The ``\\n``, ``\\r``, and ``"`` characters are
+    percent encoded.
 
-    .. _HTML5 Working Draft Section 4.10.22.7:
-        https://w3c.github.io/html/sec-forms.html#multipart-form-data
+    .. _WHATWG HTML Standard:
+        https://html.spec.whatwg.org/multipage/
+        form-control-infrastructure.html#multipart-form-data
 
     :param name:
-        The name of the parameter, a string expected to be ASCII only.
+        The name of the parameter, an ASCII-only ``str``.
     :param value:
-        The value of the parameter, provided as ``bytes`` or `str``.
+        The value of the parameter, a ``str`` or UTF-8 encoded
+        ``bytes``.
     :returns:
-        A unicode string, stripped of troublesome characters.
+        A string ``name="value"`` with the escaped value.
+
+    .. versionchanged:: 2.0.0
+        Matches the WHATWG HTML Standard as of 2021/06/10. Control
+        characters are no longer percent encoded.
+
+    .. versionchanged:: 2.0.0
+        Renamed from ``format_header_param_html5`` and
+        ``format_header_param``. The old names will be removed in
+        urllib3 v3.0.0.
     """
     if isinstance(value, bytes):
         value = value.decode("utf-8")
 
-    value = _replace_multiple(value, _HTML5_REPLACEMENTS)
-
+    # percent encode \n \r "
+    value = value.translate({10: "%0A", 13: "%0D", 34: "%22"})
     return f'{name}="{value}"'
 
 
-# For backwards-compatibility.
-format_header_param = format_header_param_html5
+def format_header_param_html5(name: str, value: _TYPE_FIELD_VALUE) -> str:
+    """
+    .. deprecated:: 2.0.0
+        Renamed to :func:`format_multipart_header_param`. Will be
+        removed in urllib3 v3.0.0.
+    """
+    import warnings
+
+    warnings.warn(
+        "'format_header_param_html5' has been renamed to "
+        "'format_multipart_header_param'. The old name will be "
+        "removed in urllib3 v3.0.0.",
+        DeprecationWarning,
+        stacklevel=2,
+    )
+    return format_multipart_header_param(name, value)
+
+
+def format_header_param(name: str, value: _TYPE_FIELD_VALUE) -> str:
+    """
+    .. deprecated:: 2.0.0
+        Renamed to :func:`format_multipart_header_param`. Will be
+        removed in urllib3 v3.0.0.
+    """
+    import warnings
+
+    warnings.warn(
+        "'format_header_param' has been renamed to "
+        "'format_multipart_header_param'. The old name will be "
+        "removed in urllib3 v3.0.0.",
+        DeprecationWarning,
+        stacklevel=2,
+    )
+    return format_multipart_header_param(name, value)
 
 
 class RequestField:
@@ -136,9 +166,11 @@ class RequestField:
         An optional filename of the request field. Must be unicode.
     :param headers:
         An optional dict-like object of headers to initially use for the field.
-    :param header_formatter:
-        An optional callable that is used to encode and format the headers. By
-        default, this is :func:`format_header_param_html5`.
+
+    .. versionchanged:: 2.0.0
+        The ``header_formatter`` parameter is deprecated and will
+        be removed in urllib3 v3.0.0. Override :meth:`_render_part`
+        instead.
     """
 
     def __init__(
@@ -147,9 +179,7 @@ def __init__(
         data: _TYPE_FIELD_VALUE,
         filename: Optional[str] = None,
         headers: Optional[Mapping[str, str]] = None,
-        header_formatter: Callable[
-            [str, _TYPE_FIELD_VALUE], str
-        ] = format_header_param_html5,
+        header_formatter: Optional[Callable[[str, _TYPE_FIELD_VALUE], str]] = None,
     ):
         self._name = name
         self._filename = filename
@@ -157,16 +187,27 @@ def __init__(
         self.headers: Dict[str, Optional[str]] = {}
         if headers:
             self.headers = dict(headers)
-        self.header_formatter = header_formatter
+
+        if header_formatter is not None:
+            import warnings
+
+            warnings.warn(
+                "The 'header_formatter' parameter is deprecated and "
+                "will be removed in urllib3 v3.0.0. Override the "
+                "'_render_part' method instead.",
+                DeprecationWarning,
+                stacklevel=2,
+            )
+            self.header_formatter = header_formatter
+        else:
+            self.header_formatter = format_multipart_header_param
 
     @classmethod
     def from_tuples(
         cls,
         fieldname: str,
         value: _TYPE_FIELD_VALUE_TUPLE,
-        header_formatter: Callable[
-            [str, _TYPE_FIELD_VALUE], str
-        ] = format_header_param_html5,
+        header_formatter: Optional[Callable[[str, _TYPE_FIELD_VALUE], str]] = None,
     ) -> "RequestField":
         """
         A :class:`~urllib3.fields.RequestField` factory from old-style tuple parameters.
@@ -210,15 +251,18 @@ def from_tuples(
 
     def _render_part(self, name: str, value: _TYPE_FIELD_VALUE) -> str:
         """
-        Overridable helper function to format a single header parameter. By
-        default, this calls ``self.header_formatter``.
+        Override this method to change how each multipart header
+        parameter is formatted. By default, this calls
+        :func:`format_multipart_header_param`.
 
         :param name:
-            The name of the parameter, a string expected to be ASCII only.
+            The name of the parameter, an ASCII-only ``str``.
         :param value:
-            The value of the parameter, provided as a unicode string.
-        """
+            The value of the parameter, a ``str`` or UTF-8 encoded
+            ``bytes``.
 
+        :meta public:
+        """
         return self.header_formatter(name, value)
 
     def _render_parts(


Test patch:
diff --git a/test/test_fields.py b/test/test_fields.py
--- a/test/test_fields.py
+++ b/test/test_fields.py
@@ -1,6 +1,13 @@
 import pytest
 
-from urllib3.fields import RequestField, format_header_param_rfc2231, guess_content_type
+from urllib3.fields import (
+    RequestField,
+    format_header_param,
+    format_header_param_html5,
+    format_header_param_rfc2231,
+    format_multipart_header_param,
+    guess_content_type,
+)
 
 
 class TestRequestField:
@@ -52,50 +59,52 @@ def test_render_parts(self):
         parts = field._render_parts([("name", "value"), ("filename", "value")])
         assert parts == 'name="value"; filename="value"'
 
-    def test_render_part_rfc2231_unicode(self):
-        field = RequestField(
-            "somename", "data", header_formatter=format_header_param_rfc2231
-        )
-        param = field._render_part("filename", "n\u00e4me")
-        assert param == "filename*=utf-8''n%C3%A4me"
+    @pytest.mark.parametrize(
+        ("value", "expect"),
+        [("näme", "filename*=utf-8''n%C3%A4me"), (b"name", 'filename="name"')],
+    )
+    def test_format_header_param_rfc2231_deprecated(self, value, expect):
+        with pytest.deprecated_call(match=r"urllib3 v3\.0\.0"):
+            param = format_header_param_rfc2231("filename", value)
 
-    def test_render_part_rfc2231_ascii(self):
-        field = RequestField(
-            "somename", "data", header_formatter=format_header_param_rfc2231
-        )
-        param = field._render_part("filename", b"name")
-        assert param == 'filename="name"'
+        assert param == expect
 
-    def test_render_part_html5_unicode(self):
-        field = RequestField("somename", "data")
-        param = field._render_part("filename", "n\u00e4me")
-        assert param == 'filename="n\u00e4me"'
+    def test_format_header_param_html5_deprecated(self):
+        with pytest.deprecated_call(match=r"urllib3 v3\.0\.0"):
+            param2 = format_header_param_html5("filename", "name")
 
-    def test_render_part_html5_ascii(self):
-        field = RequestField("somename", "data")
-        param = field._render_part("filename", b"name")
-        assert param == 'filename="name"'
+        with pytest.deprecated_call(match=r"urllib3 v3\.0\.0"):
+            param1 = format_header_param("filename", "name")
 
-    def test_render_part_html5_unicode_escape(self):
-        field = RequestField("somename", "data")
-        param = field._render_part("filename", "hello\\world\u0022")
-        assert param == 'filename="hello\\\\world%22"'
+        assert param1 == param2
 
-    def test_render_part_html5_unicode_with_control_character(self):
-        field = RequestField("somename", "data")
-        param = field._render_part("filename", "hello\x1A\x1B\x1C")
-        assert param == 'filename="hello%1A\x1B%1C"'
+    @pytest.mark.parametrize(
+        ("value", "expect"),
+        [
+            ("name", "name"),
+            ("näme", "näme"),
+            (b"n\xc3\xa4me", "näme"),
+            ("ski ⛷.txt", "ski ⛷.txt"),
+            ("control \x1A\x1B\x1C", "control \x1A\x1B\x1C"),
+            ("backslash \\", "backslash \\"),
+            ("quotes '\"", "quotes '%22"),
+            ("newline \n\r", "newline %0A%0D"),
+        ],
+    )
+    def test_format_multipart_header_param(self, value, expect):
+        param = format_multipart_header_param("filename", value)
+        assert param == f'filename="{expect}"'
 
-    def test_from_tuples_rfc2231(self):
-        field = RequestField.from_tuples(
-            "fieldname",
-            ("filen\u00e4me", "data"),
-            header_formatter=format_header_param_rfc2231,
-        )
+    def test_from_tuples(self):
+        field = RequestField.from_tuples("file", ("スキー旅行.txt", "data"))
         cd = field.headers["Content-Disposition"]
-        assert cd == "form-data; name=\"fieldname\"; filename*=utf-8''filen%C3%A4me"
+        assert cd == 'form-data; name="file"; filename="スキー旅行.txt"'
+
+    def test_from_tuples_rfc2231(self):
+        with pytest.deprecated_call(match=r"urllib3 v3\.0\.0"):
+            field = RequestField.from_tuples(
+                "file", ("näme", "data"), header_formatter=format_header_param_rfc2231
+            )
 
-    def test_from_tuples_html5(self):
-        field = RequestField.from_tuples("fieldname", ("filen\u00e4me", "data"))
         cd = field.headers["Content-Disposition"]
-        assert cd == 'form-data; name="fieldname"; filename="filen\u00e4me"'
+        assert cd == "form-data; name=\"file\"; filename*=utf-8''n%C3%A4me"