Repository: urllib3/urllib3

Issue:
UnicodeDecodeError in format_header_params
This issue was discussed here: https://github.com/kennethreitz/requests/issues/2639 and it seemed like the consensus was that this should be fixed in urllib3. 



Discussion:
@jonathan-s did you test that this is still a problem?

Yup, it still occurrs. 

I'd be glad to handle this - relevant code:

https://github.com/shazow/urllib3/blob/571fd737863fa0c60df24bce1fe4972719fd7ed2/urllib3/fields.py#L22-L47

Recapping the use case: this is generally coming into play during multipart/form-data uploads of files with Unicode names when using Python 2.

This isn't coming into play in Python 3; the flow there is that we're only encountering a caught UnicodeEncodeError at line 39, and no UnicodeDecodeError; this is because Python 3 strings are native unicode and only the encode operation is necessary. From there, we're skipping over lines 41 through 44, and just passing the Unicode string to encode_rfc_2231(), which builds RFC-2231-compliant ASCII strings from Unicode.

In comparison, at line 38 in Python 2, we encounter a UnicodeDecodeError. It seems like str.encode() is trying to load the bytes as an ASCII string before dumping the string to ASCII-encoded bytes. If it weren't for this, we'd also encounter a UnicodeEncodeError. If we pass both error types, then we'll encounter another UnicodeDecodeError at line 44, because we'll be again attempting the str.encode operation, inclusive of trying to decode the bytes as ASCII (not possible).

How I want to handle this:

First, I want to catch both error types at line 39.

Second, I don't actually think lines 43 and 44 are necessary - as US-ASCII is a strict subset of UTF-8, and str == bytes in Python 2, the operation isn't actually doing anything. This removes the possibility for the second UnicodeDecodeError, and lets the string pass through encode_rfc_2231(), as is the current behavior in Python 3 (treat the bytes as UTF-8, and escape them appropriately for transmission over ASCII).

Thoughts?

EDIT: Neglected to realize that some form of lines 43 and 44 are necessary to handle Python2 Unicode objects.

@haikuginger Would you be up for making a quick PR for feedback? Visualizing all the changes through written word is painful. :)


Resolving patch:
diff --git a/urllib3/fields.py b/urllib3/fields.py
--- a/urllib3/fields.py
+++ b/urllib3/fields.py
@@ -36,11 +36,11 @@ def format_header_param(name, value):
         result = '%s="%s"' % (name, value)
         try:
             result.encode('ascii')
-        except UnicodeEncodeError:
+        except (UnicodeEncodeError, UnicodeDecodeError):
             pass
         else:
             return result
-    if not six.PY3:  # Python 2:
+    if not six.PY3 and isinstance(value, six.text_type):  # Python 2:
         value = value.encode('utf-8')
     value = email.utils.encode_rfc2231(value, 'utf-8')
     value = '%s*=%s' % (name, value)


Test patch:
diff --git a/test/__init__.py b/test/__init__.py
--- a/test/__init__.py
+++ b/test/__init__.py
@@ -33,7 +33,6 @@ def setUp():
     clear_warnings()
     warnings.simplefilter('ignore', HTTPWarning)
 
-
 def onlyPy26OrOlder(test):
     """Skips this test unless you are on Python2.6.x or earlier."""
 
@@ -67,6 +66,17 @@ def wrapper(*args, **kwargs):
         return test(*args, **kwargs)
     return wrapper
 
+def onlyPy2(test):
+    """Skips this test unless you are on Python 2.x"""
+
+    @functools.wraps(test)
+    def wrapper(*args, **kwargs):
+        msg = "{name} requires Python 2.x to run".format(name=test.__name__)
+        if six.PY3:
+            raise SkipTest(msg)
+        return test(*args, **kwargs)
+    return wrapper
+
 def onlyPy3(test):
     """Skips this test unless you are on Python3.x"""
 
diff --git a/test/test_fields.py b/test/test_fields.py
--- a/test/test_fields.py
+++ b/test/test_fields.py
@@ -1,7 +1,8 @@
 import unittest
 
 from urllib3.fields import guess_content_type, RequestField
-from urllib3.packages.six import u
+from urllib3.packages.six import u, PY3
+from . import onlyPy2
 
 
 class TestRequestField(unittest.TestCase):
@@ -47,3 +48,9 @@ def test_render_part(self):
         field = RequestField('somename', 'data')
         param = field._render_part('filename', u('n\u00e4me'))
         self.assertEqual(param, "filename*=utf-8''n%C3%A4me")
+
+    @onlyPy2
+    def test_render_unicode_bytes_py2(self):
+        field = RequestField('somename', 'data')
+        param = field._render_part('filename', 'n\xc3\xa4me')
+        self.assertEqual(param, "filename*=utf-8''n%C3%A4me")