Repository: mitmproxy/mitmproxy

Issue:
flow.request.urlencoded_form does not work with Unicode parameters in Python 3
##### Steps to reproduce the problem:

1. Save the following as `test_script.py`:
```
import mitmproxy
import json

def request(flow):
  if flow.request.urlencoded_form:
    mitmproxy.ctx.log(str(flow.request.urlencoded_form))
  else:
    mitmproxy.ctx.log("No form parameters found")
```
2. Launch `mitmdump` with `mitmdump -s test_script.py`
3. Point your browser to `https://secure.splitwise.com/login`, and try to log in with any username or password (don't worry, you don't need an actual account)

The expected behavior is for `flow.request.urlencoded_form` to return a list of form parameters that includes `('utf8', '✓')` (or `(b'utf8', b'%E2%9C%93')`). Instead, `flow.request.urlencoded_form` returns no parameters at all, because it throws an exception trying to process the Unicode `✓` symbol that is included in the form.

##### Any other comments? What have you tried so far?

A semi-working fix is to update [this line](https://github.com/mitmproxy/mitmproxy/blob/master/mitmproxy/net/http/request.py#L363) to the following:
```
return tuple(mitmproxy.net.http.url.decode(str(self.content)))
```
It appears that `mitmproxy.net.http.url.decode` chokes when trying to process a parameter of the `bytes` type, but it succeeds when processing a parameter of the `str` type. Unfortunately, this also changes the type of the returned tuples from `bytes` to `str`, which probably isn't backwards-compatible with existing usage of mitmproxy. Not sure if there's an easy way to fix that – I plan to keep investigating, but if anyone has suggestions or expertise, I'd appreciate the help!

##### System information

```
Mitmproxy version: 0.19
Python version: 3.5.2
Platform: Darwin-16.1.0-x86_64-i386-64bit
SSL version: OpenSSL 1.0.2j  26 Sep 2016
Mac version: 10.12.1 ('', '', '') x86_64
```

/cc @jhottenstein


Discussion:
Thanks for the excellent report!
The root cause of this is the following:
```python3
>>> import urrlib
>>> urllib.parse.parse_qsl("foo=bar&baz=%E2%9C%93", keep_blank_values=True, errors='surrogateescape')
[('foo', 'bar'), ('baz', '✓')]

>>> urllib.parse.parse_qsl(b"foo=bar&baz=%E2%9C%93", keep_blank_values=True, errors='surrogateescape')
Traceback (most recent call last):
  File "<pyshell#4>", line 1, in <module>
    urllib.parse.parse_qsl(b"foo=bar&baz=%E2%9C%93", keep_blank_values=True, errors='surrogateescape')
  File "C:\Python35\lib\urllib\parse.py", line 628, in parse_qsl
    value = _coerce_result(value)
  File "C:\Python35\lib\urllib\parse.py", line 94, in _encode_result
    return obj.encode(encoding, errors)
UnicodeEncodeError: 'ascii' codec can't encode character '\u2713' in position 0: ordinal not in range(128)
```

This seems surprising to me, but who am I to judge. :-)

After looking around a bit, your proposal of returning text instead of bytes seems to be the way to go (see: http://stackoverflow.com/a/4073451/934719). While we could implement our own parse_qsl that just works on bytes, browsers apparently always use mutipart/form-data for binary files (see linked SO page), so text is the more elegant API anyway. If you want to send a PR that changes the default behaviour, briefly notes this in the doc string and dass a quick test, I'd be happy to merge that right away! :smiley:

Minor nitpick: If you send a PR: please use `self.content.decode()` instead of `str(self.content)` - that makes clear that there always is a strict ascii decode operation.

Thanks again!
I'll quickly take this over to have it in 1.0!

Resolving patch:
diff --git a/examples/complex/har_dump.py b/examples/complex/har_dump.py
--- a/examples/complex/har_dump.py
+++ b/examples/complex/har_dump.py
@@ -136,7 +136,7 @@ def response(flow):
 
     if flow.request.method in ["POST", "PUT", "PATCH"]:
         params = [
-            {"name": a.decode("utf8", "surrogateescape"), "value": b.decode("utf8", "surrogateescape")}
+            {"name": a, "value": b}
             for a, b in flow.request.urlencoded_form.items(multi=True)
         ]
         entry["request"]["postData"] = {
diff --git a/mitmproxy/addons/serverplayback.py b/mitmproxy/addons/serverplayback.py
--- a/mitmproxy/addons/serverplayback.py
+++ b/mitmproxy/addons/serverplayback.py
@@ -1,9 +1,10 @@
-import urllib
 import hashlib
+import urllib
+from typing import Any  # noqa
+from typing import List  # noqa
 
-from mitmproxy.utils import strutils
-from mitmproxy import exceptions
 from mitmproxy import ctx
+from mitmproxy import exceptions
 from mitmproxy import io
 
 
@@ -36,17 +37,20 @@ def _hash(self, flow):
         _, _, path, _, query, _ = urllib.parse.urlparse(r.url)
         queriesArray = urllib.parse.parse_qsl(query, keep_blank_values=True)
 
-        key = [str(r.port), str(r.scheme), str(r.method), str(path)]
+        key = [str(r.port), str(r.scheme), str(r.method), str(path)]  # type: List[Any]
         if not self.options.server_replay_ignore_content:
-            form_contents = r.urlencoded_form or r.multipart_form
-            if self.options.server_replay_ignore_payload_params and form_contents:
-                params = [
-                    strutils.always_bytes(i)
-                    for i in self.options.server_replay_ignore_payload_params
-                ]
-                for p in form_contents.items(multi=True):
-                    if p[0] not in params:
-                        key.append(p)
+            if self.options.server_replay_ignore_payload_params and r.multipart_form:
+                key.extend(
+                    (k, v)
+                    for k, v in r.multipart_form.items(multi=True)
+                    if k.decode(errors="replace") not in self.options.server_replay_ignore_payload_params
+                )
+            elif self.options.server_replay_ignore_payload_params and r.urlencoded_form:
+                key.extend(
+                    (k, v)
+                    for k, v in r.urlencoded_form.items(multi=True)
+                    if k not in self.options.server_replay_ignore_payload_params
+                )
             else:
                 key.append(str(r.raw_content))
 
diff --git a/mitmproxy/net/http/request.py b/mitmproxy/net/http/request.py
--- a/mitmproxy/net/http/request.py
+++ b/mitmproxy/net/http/request.py
@@ -350,6 +350,8 @@ def urlencoded_form(self):
         The URL-encoded form data as an :py:class:`~mitmproxy.net.multidict.MultiDictView` object.
         An empty multidict.MultiDictView if the content-type indicates non-form data
         or the content could not be parsed.
+
+        Starting with mitmproxy 1.0, key and value are strings.
         """
         return multidict.MultiDictView(
             self._get_urlencoded_form,
@@ -360,7 +362,7 @@ def _get_urlencoded_form(self):
         is_valid_content_type = "application/x-www-form-urlencoded" in self.headers.get("content-type", "").lower()
         if is_valid_content_type:
             try:
-                return tuple(mitmproxy.net.http.url.decode(self.content))
+                return tuple(mitmproxy.net.http.url.decode(self.content.decode()))
             except ValueError:
                 pass
         return ()
@@ -381,7 +383,10 @@ def urlencoded_form(self, value):
     def multipart_form(self):
         """
         The multipart form data as an :py:class:`~mitmproxy.net.multidict.MultiDictView` object.
-        None if the content-type indicates non-form data.
+        An empty multidict.MultiDictView if the content-type indicates non-form data
+        or the content could not be parsed.
+
+        Key and value are bytes.
         """
         return multidict.MultiDictView(
             self._get_multipart_form,


Test patch:
diff --git a/test/mitmproxy/net/http/test_request.py b/test/mitmproxy/net/http/test_request.py
--- a/test/mitmproxy/net/http/test_request.py
+++ b/test/mitmproxy/net/http/test_request.py
@@ -255,11 +255,11 @@ def test_get_urlencoded_form(self):
         assert not request.urlencoded_form
 
         request.headers["Content-Type"] = "application/x-www-form-urlencoded"
-        assert list(request.urlencoded_form.items()) == [(b"foobar", b"baz")]
+        assert list(request.urlencoded_form.items()) == [("foobar", "baz")]
 
     def test_set_urlencoded_form(self):
         request = treq()
-        request.urlencoded_form = [(b'foo', b'bar'), (b'rab', b'oof')]
+        request.urlencoded_form = [('foo', 'bar'), ('rab', 'oof')]
         assert request.headers["Content-Type"] == "application/x-www-form-urlencoded"
         assert request.content
 
diff --git a/test/mitmproxy/test_examples.py b/test/mitmproxy/test_examples.py
--- a/test/mitmproxy/test_examples.py
+++ b/test/mitmproxy/test_examples.py
@@ -68,11 +68,11 @@ def test_modify_form(self):
         f = tflow.tflow(req=tutils.treq(headers=form_header))
         m.request(f)
 
-        assert f.request.urlencoded_form[b"mitmproxy"] == b"rocks"
+        assert f.request.urlencoded_form["mitmproxy"] == "rocks"
 
         f.request.headers["content-type"] = ""
         m.request(f)
-        assert list(f.request.urlencoded_form.items()) == [(b"foo", b"bar")]
+        assert list(f.request.urlencoded_form.items()) == [("foo", "bar")]
 
     def test_modify_querystring(self):
         m, sc = tscript("simple/modify_querystring.py")