Repository: urllib3/urllib3

Issue:
Consider HTML 5 draft for multipart/form-data
In several past posts, I've tried to make urllib3 more standards-compliant, the way I saw things. In particular, #120 aimed at adding a proper `Content-Type` header for _every_ form field, and #119 / #223 introduced [RFC 2231](http://tools.ietf.org/html/rfc2231) format for file name encoding.

One consequence of that last change was my noticing that tornado doesn't decode that format, causing some test cases to fail. e11e03628f8b82021a74ba7f5b39ce9ad9011382 monkey-patched tornado to deal with that, and I filed facebook/tornado#868 to take this upstream. There [bdarnell pointed out](https://github.com/facebook/tornado/pull/869#issuecomment-23632083) the following:

**The current HTML 5 draft has [a section on multipart/form-data](http://www.w3.org/html/wg/drafts/html/master/forms.html#multipart-form-data) which explicitely forbids the use of `Content-Type` for non-file fields, and also the use of RFC 2231 format for file names. It also provides its own mechanism for encoding field names.**

Although I'm reluctant to abandon conformance with Standards Track RFCs for the sake of some lines in a draft, it appears that the HTML 5 draft might better reflect what current web servers actually implement, or are going to implement in the intermediate future. Perople have already [reported problems](http://stackoverflow.com/q/20591599/1468366) with our format in the wild. For this reason, I wonder whether the current choice of always using RFC 2231 might have to be revised.

I can imagine two approaches, one would be dropping RFC 2231 support completely, the other would be introducing some switch to distinguish cases. I guess (although I don't like it) that the switch should default to the HTML 5 way. I guess I'd implement the switch as a global variable, since every field needs acces to it and concurrent use of multiple standards should be very unlikely. Do you have a better suggestion?

So what would be needed for HTML 5 conforming handling of encodings? We'd need to properly distinguish between files and non-files. The [`RequestField` class](https://github.com/shazow/urllib3/blob/master/urllib3/fields.py#L53) could probably check whether both file name and content type are `None` to recognize non-file fields. That choice could be used to control the addition of the `Content-Type` header.

For file names, we'd have to encode them to UTF-8 (or some user-configurable request encoding?), after making sure the result does not contain any invalid data: newlines must be stripped (or converted to spaces), quotation marks and backslashes quoted by `\` or substituted. I'm not sure which of these, and how much control we want to give our users over that process.

If we allow for encodings other than UTF-8, which are not able to represent all Unicode codepoints, then we might need some more work. File name characters which are not representable must then be substituted or approximated somehow. For filed names we'd have to make sure we replace non-representable characters as XML decimal character entities instead. Other non-ASCII header fields should usually not occur, but if they do, I guess they should be treated the same way file names are.

I notice that I made a mistake in reading [RFC 2388](http://tools.ietf.org/html/rfc2388): its [section 5.4](http://tools.ietf.org/html/rfc2388#section-5.4) states that field names are subject to a different encoding than file names. Grrr! Whoever though of that?!? In any case, to be conforming to that as well, we might even want to revise the RFC2231-using mode if we decide to keep it at all.

I might find the time to write some of this myself, but before I start, I'd like to discuss the open questions above as to how this should get implemented.
## 



Discussion:
Oh boy. First, my sincerest thanks for your thorough research and presenting of the situation. :) As I work through my backlog of emails, please excuse and remind me if I miss an important detail. The questions I noticed:
- Should we use a global variable to switch between "RFC 2231" and "HTML 5" modes?
  
  -1 on this. Whenever possible, urllib3 tries to be configurable per-request so that you could execute multiple different libraries using different features/states of urllib3 without a conflicting global state.
- Should we abandon RFC 2231 altogether in favour of HTML 5's spec?
  
  -1 on this too. I'd like to at least continue to support as much customizability of the request as possible, even if the functionality is a superset of an HTTP spec. That said, it may be wise to assume a conservative default (seems to be HTML 5 in this case).
- What about distinguishing between files and non-files in the `RequestField` class?
  
  +1, that sounds perfect.
- <Stuff about encoding that I don't quite understand>
  
  Honestly I'm not sure what's the Right Thing to do here. UTF8 does sound like the most sensible Python thing to do, but about other normalization... I'm inclined to start with a helper that lives in `urllib3.util` and see where we want to go from there. I'm reluctant to support anything other than UTF8 by default, but we should not get in the way if people want to encode things themselves.

Did I miss anything?

I would be more than happy to continue having you lead this if you're up for it. :)

Have a look at [my current work in progress](https://github.com/gagern/urllib3/compare/issue303a), and let me know whether it's going in the right direction. I've managed to pass the request object along, so that things can be configured there. The new attributes which will control behavior still need to be documented somewhere, please let me know where you'd like to see them. Should I file a pull request to discuss the code, or wait till I consider it ready for merging (i.e. docs added)?

With respect to that encoding issue: I've also introduced a request parameter to control encoding, set to UTF-8 by default. The main reason why someone might want to change that is because some HTML 5 web form without accept-charset attribute was part of a HTML file with charset other than UTF-8. In that case, HTML 5 requires using that charset, so urllib3 should provide some means to do so. Manually encoding field bodies would be part of the solution, but would not work for headers like file names which have to be Unicode for our mechanisms to work. Now everything is encoded using a configurable encoding, with xml character references as substitution, just like HTML 5 specifies.

@gagern Yes, please open a PR so we can continue the discussion there and I can leave comments on the diff. :)

The above summary and links were very helpful in understading the situation, thanks! Unfortunately, there still isn't a single, clear, widely adopted way to encode non-ascii file names, and web servers don't agree on what format they accept. It's a fairly simple monkeypatch to workaround if you're interacting with a single web server, but there is still a need to be able to change encoding formats per-request. @shazow @gagern Is any one working on this (or even wants to)? 

Also, here is an unbroken link to the relevant section in the current HTML5 draft: [multipart-form-data](http://w3c.github.io/html/sec-forms.html#multipart-form-data)

I'm not aware of anyone working on it, so you're welcome to claim it. :)


Resolving patch:
diff --git a/dummyserver/handlers.py b/dummyserver/handlers.py
--- a/dummyserver/handlers.py
+++ b/dummyserver/handlers.py
@@ -17,6 +17,7 @@
 
 from urllib3.packages.six.moves.http_client import responses
 from urllib3.packages.six.moves.urllib.parse import urlsplit
+from urllib3.packages.six import binary_type
 
 log = logging.getLogger(__name__)
 
@@ -157,10 +158,15 @@ def upload(self, request):
             return Response("Wrong size: %d != %d" %
                             (size, len(data)), status='400 Bad Request')
 
-        if filename != file_['filename']:
-            return Response("Wrong filename: %s != %s" %
-                            (filename, file_.filename),
-                            status='400 Bad Request')
+        got_filename = file_['filename']
+        if(isinstance(got_filename, binary_type)):
+            got_filename = got_filename.decode('utf-8')
+
+        # Tornado can leave the trailing \n in place on the filename.
+        if filename != got_filename:
+            return Response(
+                u"Wrong filename: %s != %s" % (filename, file_.filename),
+                status='400 Bad Request')
 
         return Response()
 
@@ -304,49 +310,3 @@ def redirect_after(self, request):
 
     def shutdown(self, request):
         sys.exit()
-
-
-# RFC2231-aware replacement of internal tornado function
-def _parse_header(line):
-    r"""Parse a Content-type like header.
-
-    Return the main content-type and a dictionary of options.
-
-    >>> d = _parse_header("CD: fd; foo=\"bar\"; file*=utf-8''T%C3%A4st")[1]
-    >>> d['file'] == 'T\u00e4st'
-    True
-    >>> d['foo']
-    'bar'
-    """
-    import tornado.httputil
-    import email.utils
-    from urllib3.packages import six
-    if not six.PY3:
-        line = line.encode('utf-8')
-    parts = tornado.httputil._parseparam(';' + line)
-    key = next(parts)
-    # decode_params treats first argument special, but we already stripped key
-    params = [('Dummy', 'value')]
-    for p in parts:
-        i = p.find('=')
-        if i >= 0:
-            name = p[:i].strip().lower()
-            value = p[i + 1:].strip()
-            params.append((name, value))
-    params = email.utils.decode_params(params)
-    params.pop(0)  # get rid of the dummy again
-    pdict = {}
-    for name, value in params:
-        value = email.utils.collapse_rfc2231_value(value)
-        if len(value) >= 2 and value[0] == '"' and value[-1] == '"':
-            value = value[1:-1]
-        pdict[name] = value
-    return key, pdict
-
-
-# TODO: make the following conditional as soon as we know a version
-#       which does not require this fix.
-#       See https://github.com/facebook/tornado/issues/868
-if True:
-    import tornado.httputil
-    tornado.httputil._parse_header = _parse_header
diff --git a/src/urllib3/fields.py b/src/urllib3/fields.py
--- a/src/urllib3/fields.py
+++ b/src/urllib3/fields.py
@@ -1,6 +1,7 @@
 from __future__ import absolute_import
 import email.utils
 import mimetypes
+import re
 
 from .packages import six
 
@@ -19,57 +20,147 @@ def guess_content_type(filename, default='application/octet-stream'):
     return default
 
 
-def format_header_param(name, value):
+def format_header_param_rfc2231(name, value):
     """
-    Helper function to format and quote a single header parameter.
+    Helper function to format and quote a single header parameter using the
+    strategy defined in RFC 2231.
 
     Particularly useful for header parameters which might contain
-    non-ASCII values, like file names. This follows RFC 2231, as
-    suggested by RFC 2388 Section 4.4.
+    non-ASCII values, like file names. This follows RFC 2388 Section 4.4.
 
     :param name:
         The name of the parameter, a string expected to be ASCII only.
     :param value:
-        The value of the parameter, provided as a unicode string.
+        The value of the parameter, provided as ``bytes`` or `str``.
+    :ret:
+        An RFC-2231-formatted unicode string.
     """
+    if isinstance(value, six.binary_type):
+        value = value.decode("utf-8")
+
     if not any(ch in value for ch in '"\\\r\n'):
-        result = '%s="%s"' % (name, value)
+        result = u'%s="%s"' % (name, value)
         try:
             result.encode('ascii')
         except (UnicodeEncodeError, UnicodeDecodeError):
             pass
         else:
             return result
-    if not six.PY3 and isinstance(value, six.text_type):  # Python 2:
+
+    if not six.PY3:  # Python 2:
         value = value.encode('utf-8')
+
+    # encode_rfc2231 accepts an encoded string and returns an ascii-encoded
+    # string in Python 2 but accepts and returns unicode strings in Python 3
     value = email.utils.encode_rfc2231(value, 'utf-8')
     value = '%s*=%s' % (name, value)
+
+    if not six.PY3:  # Python 2:
+        value = value.decode('utf-8')
+
     return value
 
 
+_HTML5_REPLACEMENTS = {
+    u"\u0022": u"%22",
+    # Replace "\" with "\\".
+    u"\u005C": u"\u005C\u005C",
+    u"\u005C": u"\u005C\u005C",
+}
+
+# All control characters from 0x00 to 0x1F *except* 0x1B.
+_HTML5_REPLACEMENTS.update({
+    six.unichr(cc): u"%{:02X}".format(cc)
+    for cc
+    in range(0x00, 0x1F+1)
+    if cc not in (0x1B,)
+})
+
+
+def _replace_multiple(value, needles_and_replacements):
+
+    def replacer(match):
+        return needles_and_replacements[match.group(0)]
+
+    pattern = re.compile(
+        r"|".join([
+            re.escape(needle) for needle in needles_and_replacements.keys()
+        ])
+    )
+
+    result = pattern.sub(replacer, value)
+
+    return result
+
+
+def format_header_param_html5(name, value):
+    """
+    Helper function to format and quote a single header parameter using the
+    HTML5 strategy.
+
+    Particularly useful for header parameters which might contain
+    non-ASCII values, like file names. This follows the `HTML5 Working Draft
+    Section 4.10.22.7`_ and matches the behavior of curl and modern browsers.
+
+    .. _HTML5 Working Draft Section 4.10.22.7:
+        https://w3c.github.io/html/sec-forms.html#multipart-form-data
+
+    :param name:
+        The name of the parameter, a string expected to be ASCII only.
+    :param value:
+        The value of the parameter, provided as ``bytes`` or `str``.
+    :ret:
+        A unicode string, stripped of troublesome characters.
+    """
+    if isinstance(value, six.binary_type):
+        value = value.decode("utf-8")
+
+    value = _replace_multiple(value, _HTML5_REPLACEMENTS)
+
+    return u'%s="%s"' % (name, value)
+
+
+# For backwards-compatibility.
+format_header_param = format_header_param_html5
+
+
 class RequestField(object):
     """
     A data container for request body parameters.
 
     :param name:
-        The name of this request field.
+        The name of this request field. Must be unicode.
     :param data:
         The data/value body.
     :param filename:
-        An optional filename of the request field.
+        An optional filename of the request field. Must be unicode.
     :param headers:
         An optional dict-like object of headers to initially use for the field.
+    :param header_formatter:
+        An optional callable that is used to encode and format the headers. By
+        default, this is :func:`format_header_param_html5`.
     """
-    def __init__(self, name, data, filename=None, headers=None):
+    def __init__(
+            self,
+            name,
+            data,
+            filename=None,
+            headers=None,
+            header_formatter=format_header_param_html5):
         self._name = name
         self._filename = filename
         self.data = data
         self.headers = {}
         if headers:
             self.headers = dict(headers)
+        self.header_formatter = header_formatter
 
     @classmethod
-    def from_tuples(cls, fieldname, value):
+    def from_tuples(
+            cls,
+            fieldname,
+            value,
+            header_formatter=format_header_param_html5):
         """
         A :class:`~urllib3.fields.RequestField` factory from old-style tuple parameters.
 
@@ -97,21 +188,24 @@ def from_tuples(cls, fieldname, value):
             content_type = None
             data = value
 
-        request_param = cls(fieldname, data, filename=filename)
+        request_param = cls(
+            fieldname, data, filename=filename, header_formatter=header_formatter)
         request_param.make_multipart(content_type=content_type)
 
         return request_param
 
     def _render_part(self, name, value):
         """
-        Overridable helper function to format a single header parameter.
+        Overridable helper function to format a single header parameter. By
+        default, this calls ``self.header_formatter``.
 
         :param name:
             The name of the parameter, a string expected to be ASCII only.
         :param value:
             The value of the parameter, provided as a unicode string.
         """
-        return format_header_param(name, value)
+
+        return self.header_formatter(name, value)
 
     def _render_parts(self, header_parts):
         """
@@ -133,7 +227,7 @@ def _render_parts(self, header_parts):
             if value is not None:
                 parts.append(self._render_part(name, value))
 
-        return '; '.join(parts)
+        return u'; '.join(parts)
 
     def render_headers(self):
         """
@@ -144,15 +238,15 @@ def render_headers(self):
         sort_keys = ['Content-Disposition', 'Content-Type', 'Content-Location']
         for sort_key in sort_keys:
             if self.headers.get(sort_key, False):
-                lines.append('%s: %s' % (sort_key, self.headers[sort_key]))
+                lines.append(u'%s: %s' % (sort_key, self.headers[sort_key]))
 
         for header_name, header_value in self.headers.items():
             if header_name not in sort_keys:
                 if header_value:
-                    lines.append('%s: %s' % (header_name, header_value))
+                    lines.append(u'%s: %s' % (header_name, header_value))
 
-        lines.append('\r\n')
-        return '\r\n'.join(lines)
+        lines.append(u'\r\n')
+        return u'\r\n'.join(lines)
 
     def make_multipart(self, content_disposition=None, content_type=None,
                        content_location=None):
@@ -168,10 +262,10 @@ def make_multipart(self, content_disposition=None, content_type=None,
             The 'Content-Location' of the request body.
 
         """
-        self.headers['Content-Disposition'] = content_disposition or 'form-data'
-        self.headers['Content-Disposition'] += '; '.join([
-            '', self._render_parts(
-                (('name', self._name), ('filename', self._filename))
+        self.headers['Content-Disposition'] = content_disposition or u'form-data'
+        self.headers['Content-Disposition'] += u'; '.join([
+            u'', self._render_parts(
+                ((u'name', self._name), (u'filename', self._filename))
             )
         ])
         self.headers['Content-Type'] = content_type


Test patch:
diff --git a/test/test_fields.py b/test/test_fields.py
--- a/test/test_fields.py
+++ b/test/test_fields.py
@@ -1,8 +1,7 @@
 import pytest
 
-from urllib3.fields import guess_content_type, RequestField
+from urllib3.fields import format_header_param_rfc2231, guess_content_type, RequestField
 from urllib3.packages.six import u
-from . import onlyPy2
 
 
 class TestRequestField(object):
@@ -53,13 +52,45 @@ def test_render_parts(self):
         parts = field._render_parts([('name', 'value'), ('filename', 'value')])
         assert parts == 'name="value"; filename="value"'
 
-    def test_render_part(self):
-        field = RequestField('somename', 'data')
+    def test_render_part_rfc2231_unicode(self):
+        field = RequestField('somename', 'data', header_formatter=format_header_param_rfc2231)
         param = field._render_part('filename', u('n\u00e4me'))
         assert param == "filename*=utf-8''n%C3%A4me"
 
-    @onlyPy2
-    def test_render_unicode_bytes_py2(self):
+    def test_render_part_rfc2231_ascii(self):
+        field = RequestField('somename', 'data', header_formatter=format_header_param_rfc2231)
+        param = field._render_part('filename', b'name')
+        assert param == 'filename="name"'
+
+    def test_render_part_html5_unicode(self):
         field = RequestField('somename', 'data')
-        param = field._render_part('filename', 'n\xc3\xa4me')
-        assert param == "filename*=utf-8''n%C3%A4me"
+        param = field._render_part('filename', u('n\u00e4me'))
+        assert param == u('filename="n\u00e4me"')
+
+    def test_render_part_html5_ascii(self):
+        field = RequestField('somename', 'data')
+        param = field._render_part('filename', b'name')
+        assert param == 'filename="name"'
+
+    def test_render_part_html5_unicode_escape(self):
+        field = RequestField('somename', 'data')
+        param = field._render_part('filename', u('hello\\world\u0022'))
+        assert param == u('filename="hello\\\\world%22"')
+
+    def test_render_part_html5_unicode_with_control_character(self):
+        field = RequestField('somename', 'data')
+        param = field._render_part('filename', u('hello\x1A\x1B\x1C'))
+        assert param == u('filename="hello%1A\x1B%1C"')
+
+    def test_from_tuples_rfc2231(self):
+        field = RequestField.from_tuples(
+            u('fieldname'),
+            (u('filen\u00e4me'), 'data'),
+            header_formatter=format_header_param_rfc2231)
+        cd = field.headers['Content-Disposition']
+        assert (cd == u("form-data; name=\"fieldname\"; filename*=utf-8''filen%C3%A4me"))
+
+    def test_from_tuples_html5(self):
+        field = RequestField.from_tuples(u('fieldname'), (u('filen\u00e4me'), 'data'))
+        cd = field.headers['Content-Disposition']
+        assert (cd == u('form-data; name="fieldname"; filename="filen\u00e4me"'))