Repository: mitmproxy/mitmproxy

Issue:
url with Chinese query string return 400
use IE11,url with Chinese query string,return 400.

1. Mitmproxy version: 3.0.0 (release version)
    Python version: 3.5.3
    Platform: Windows-10-10.0.14393-SP0
    SSL version: OpenSSL 1.1.0e 16 Feb 2017 
    Windows version: 10 10.0.14393 SP0 Multiprocessor Free
2. chrome+mitmdump is fine.
3. but use IE11+mitmdump is error.  
4. use IE11 + burpsuite is fine.
5. mitmdump --listen-host 127.0.0.1 --listen-port 8080

Mitmproxy was no hint error, but query string **lc_name** was submitted to the charset difference.
return HTTP 400.
html charset is gb2312.

IE11 developer tools see url   http://wlpx.tax-edu.net/jsp/portal/PortalControl?flag=selectCourse&lc_id=42825&lc_name=é��ç�¨ç�¥è¯�å��è�½å��

chrome developer tools see  url http://wlpx.tax-edu.net/jsp/portal/PortalControl?flag=selectCourse&lc_id=42825&lc_name=%CD%A8%D3%C3%D6%AA%CA%B6%BA%CD%C4%DC%C1%A6

![image](https://cloud.githubusercontent.com/assets/1232444/25463418/abe2ef0e-2b28-11e7-827c-ea73b525213c.png)





Discussion:
Thanks for the report.

For reference, here is what I believe is the original URL: 

```
>>> x = "http://wlpx.tax-edu.net/jsp/portal/PortalControl?flag=selectCourse&lc_id=42825&lc_name=茅莽莽猫氓猫氓"
>>> x.encode("gb2312")
b"http://wlpx.tax-edu.net/jsp/portal/PortalControl?flag=selectCourse&lc_id=42825&lc_name=\xc3\xa9\xc3\xa7\xc3\xa7\xc3\xa8\xc3\xa5\xc3\xa8\xc3\xa5"
```

Unfortunately, there is no encoding provided and this would also be valid UTF-8:

```
>>> x.encode("gb2312").decode("utf8")
'http://wlpx.tax-edu.net/jsp/portal/PortalControl?flag=selectCourse&lc_id=42825&lc_name=éççèåèå'
```

Whatsoever, looking at https://blogs.msdn.microsoft.com/ieinternals/2012/07/13/brain-dump-international-text/, we should probably just treat it as an opaque collection of bytes.

-----------

Non-IE repro for the original bug:

```
λ curl -x localhost:8080 -k 'http://wlpx.tax-edu.net/jsp/portal/PortalControl?flag=selectCourse&lc_id=42825&lc_name=éççèåèå'
```
```
127.0.0.1:53226: HTTP protocol error in client request: Bad HTTP request line: b'GET http://wlpx.tax-edu.net/jsp/portal/PortalControl?flag=selectCourse&lc_id=42825&lc_name=\xc3\xa9\xc3\xa7\xc3\xa7\xc3\xa8\xc3\xa5\xc3\xa8\xc3\xa5 HTTP/1.1'
```
thanks.
uncheck ie11 send utf-8 urls Option.
mitmproxy is same as the curl.
127.0.0.1:50800: HTTP protocol error in client request: Bad HTTP request line: b'GET http://wlpx.tax-edu.net/jsp/portal/PortalControl?flag=selectCourse&lc_id=42825&lc_name=\xcd\xa8\xd3\xc3\xd6\xaa\xca\xb6\xba\xcd\xc4\xdc\xc1\xa6 HTTP/1.1'

Ok, the root cause is that we call `urllib.parse.urlparse` [here](https://github.com/mitmproxy/mitmproxy/blob/40f387eb48c192391fcf265e05d0605217b58ea7/mitmproxy/net/http/url.py#L40), which raises a ValueError for the following:
```
>>> import urllib
>>> urllib.parse.urlparse(b'http://wlpx.tax-edu.net/jsp/portal/PortalControl?flag=selectCourse&lc_id=42825&lc_name=\xc3\xa9\xc3\xa7\xc3\xa7\xc3\xa8\xc3\xa5\xc3\xa8\xc3\xa5')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 87: ordinal not in range(128)
```

From https://docs.python.org/3/library/urllib.parse.html:
> Applications that need to operate on potentially improperly quoted URLs that may contain non-ASCII data will need to do their own decoding from bytes to characters before invoking the URL parsing methods.


decoding from bytes  to characters.
How do you know what it is encoding charset?
Well, yes... that's problem - we simply don't know and there's no information provided.
can import chardet?
Can we workaround this by detecting non-ascii characters in url and then using urrlib.parse.quote to quote/escape the characters?

Resolving patch:
diff --git a/mitmproxy/net/http/url.py b/mitmproxy/net/http/url.py
--- a/mitmproxy/net/http/url.py
+++ b/mitmproxy/net/http/url.py
@@ -21,16 +21,25 @@ def parse(url):
         Raises:
             ValueError, if the URL is not properly formatted.
     """
-    parsed = urllib.parse.urlparse(url)
+    # Size of Ascii character after encoding is 1 byte which is same as its size
+    # But non-Ascii character's size after encoding will be more than its size
+    def ascii_check(l):
+        if len(l) == len(str(l).encode()):
+            return True
+        return False
+
+    if isinstance(url, bytes):
+        url = url.decode()
+        if not ascii_check(url):
+            url = urllib.parse.urlsplit(url)
+            url = list(url)
+            url[3] = urllib.parse.quote(url[3])
+            url = urllib.parse.urlunsplit(url)
 
+    parsed = urllib.parse.urlparse(url)
     if not parsed.hostname:
         raise ValueError("No hostname given")
 
-    if isinstance(url, bytes):
-        host = parsed.hostname
-
-        # this should not raise a ValueError,
-        # but we try to be very forgiving here and accept just everything.
     else:
         host = parsed.hostname.encode("idna")
         if isinstance(parsed, urllib.parse.ParseResult):


Test patch:
diff --git a/test/mitmproxy/net/http/test_url.py b/test/mitmproxy/net/http/test_url.py
--- a/test/mitmproxy/net/http/test_url.py
+++ b/test/mitmproxy/net/http/test_url.py
@@ -49,6 +49,17 @@ def test_parse():
         url.parse('http://lo[calhost')
 
 
+def test_ascii_check():
+
+    test_url = "https://xyz.tax-edu.net?flag=selectCourse&lc_id=42825&lc_name=茅莽莽猫氓猫氓".encode()
+    scheme, host, port, full_path = url.parse(test_url)
+    assert scheme == b'https'
+    assert host == b'xyz.tax-edu.net'
+    assert port == 443
+    assert full_path == b'/?flag%3DselectCourse%26lc_id%3D42825%26lc_name%3D%E8%8C%85%E8%8E%BD%E8%8E' \
+                        b'%BD%E7%8C%AB%E6%B0%93%E7%8C%AB%E6%B0%93'
+
+
 @pytest.mark.skipif(sys.version_info < (3, 6), reason='requires Python 3.6 or higher')
 def test_parse_port_range():
     # Port out of range
@@ -61,6 +72,7 @@ def test_unparse():
     assert url.unparse("http", "foo.com", 80, "/bar") == "http://foo.com/bar"
     assert url.unparse("https", "foo.com", 80, "") == "https://foo.com:80"
     assert url.unparse("https", "foo.com", 443, "") == "https://foo.com"
+    assert url.unparse("https", "foo.com", 443, "*") == "https://foo.com"
 
 
 # We ignore the byte 126: '~' because of an incompatibility in Python 3.6 and 3.7
@@ -131,3 +143,7 @@ def test_unquote():
     assert url.unquote("foo") == "foo"
     assert url.unquote("foo%20bar") == "foo bar"
     assert url.unquote(surrogates_quoted) == surrogates
+
+
+def test_hostport():
+    assert url.hostport(b"https", b"foo.com", 8080) == b"foo.com:8080"