Repository: aio-libs/aiohttp

Issue:
Not work with url with ':'
Hi, 

get trouble with use aiohttp:
http://www.homeaway.com/search/refined/world/region:1/Leisure+Activities:golf
don't give correct answer. urllib.request work correct

regards,
V



Discussion:
Hi @pnvasko,
Can you be more specific? How exactly aiohttp does not provide correct answer?
I've just tried and got `200 ok` with body about 30Kb.

Change target url. For sample ask - http://www.homeaway.com/search/refined/world/region:1/Leisure+Activities:golf, get content like this page  http://www.homeaway.com/search

ok, I see.
Actually the problem is with `+` it gets quoted into `%2B`

What can I do ?

понедельник, 31 августа 2015 г. пользователь Alexey Popravka написал:

> ok, I see.
> Actually the problem is with + it gets quoted into %2B
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/KeepSafe/aiohttp/issues/480#issuecomment-136363210.

@fafhrd91, @asvetlov guys!
The problem is here [aiohttp/client_reqrep.py:178](https://github.com/KeepSafe/aiohttp/blob/master/aiohttp/client_reqrep.py#L178)
I've 'blamed' that line to the very begining (https://github.com/KeepSafe/aiohttp/commit/f382b5ffc445e45a110734f5396728da7914aeb6)
and I could not see why `path` should be re-quoted.
The `urlsplit` returns path as is, so if it was already quoted it gets double-quoted.
And it also leads to re-quoting of `location` header in redirects.

I think we should remove this quoting of path and leave it up to end-users.
What do you think?

I think we should make trivial one-page app displaying `request.raw_path` then open the page in browser and play with different characters in path string.

Reference to standard is also good but I like to see what does Chrome and Firefox in case of  suspicious symbols


Resolving patch:
diff --git a/aiohttp/client_reqrep.py b/aiohttp/client_reqrep.py
--- a/aiohttp/client_reqrep.py
+++ b/aiohttp/client_reqrep.py
@@ -174,8 +174,8 @@ def update_path(self, params):
             else:
                 query = params
 
-        self.path = urllib.parse.urlunsplit(
-            ('', '', urllib.parse.quote(path, safe='/%:='), query, fragment))
+        self.path = urllib.parse.urlunsplit(('', '', helpers.requote_uri(path),
+                                             query, fragment))
         self.url = urllib.parse.urlunsplit(
             (scheme, netloc, self.path, '', ''))
 
diff --git a/aiohttp/errors.py b/aiohttp/errors.py
--- a/aiohttp/errors.py
+++ b/aiohttp/errors.py
@@ -186,3 +186,7 @@ def __repr__(self):
         return '<{} expected={} got={} host={} port={}>'.format(
             self.__class__.__name__, self.expected, self.got,
             self.host, self.port)
+
+
+class InvalidURL(Exception):
+    """Invalid URL."""
diff --git a/aiohttp/helpers.py b/aiohttp/helpers.py
--- a/aiohttp/helpers.py
+++ b/aiohttp/helpers.py
@@ -2,11 +2,12 @@
 import base64
 import io
 import os
-import urllib.parse
+from urllib.parse import quote, urlencode
 from collections import namedtuple
 from wsgiref.handlers import format_date_time
 
 from . import hdrs, multidict
+from .errors import InvalidURL
 
 __all__ = ('BasicAuth', 'FormData', 'parse_mimetype')
 
@@ -126,7 +127,7 @@ def _gen_form_urlencoded(self, encoding):
         for type_options, _, value in self._fields:
             data.append((type_options['name'], value))
 
-        data = urllib.parse.urlencode(data, doseq=True)
+        data = urlencode(data, doseq=True)
         return data.encode(encoding)
 
     def _gen_form_data(self, *args, **kwargs):
@@ -322,3 +323,51 @@ def __get__(self, inst, owner, _marker=_marker):
 
     def __set__(self, inst, value):
         raise AttributeError("reified property is read-only")
+
+
+# The unreserved URI characters (RFC 3986)
+UNRESERVED_SET = frozenset(
+    "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
+    + "0123456789-._~")
+
+
+def unquote_unreserved(uri):
+    """Un-escape any percent-escape sequences in a URI that are unreserved
+    characters. This leaves all reserved, illegal and non-ASCII bytes encoded.
+    """
+    parts = uri.split('%')
+    for i in range(1, len(parts)):
+        h = parts[i][0:2]
+        if len(h) == 2 and h.isalnum():
+            try:
+                c = chr(int(h, 16))
+            except ValueError:
+                raise InvalidURL("Invalid percent-escape sequence: '%s'" % h)
+
+            if c in UNRESERVED_SET:
+                parts[i] = c + parts[i][2:]
+            else:
+                parts[i] = '%' + parts[i]
+        else:
+            parts[i] = '%' + parts[i]
+    return ''.join(parts)
+
+
+def requote_uri(uri):
+    """Re-quote the given URI.
+
+    This function passes the given URI through an unquote/quote cycle to
+    ensure that it is fully and consistently quoted.
+    """
+    safe_with_percent = "!#$%&'()*+,/:;=?@[]~"
+    safe_without_percent = "!#$&'()*+,/:;=?@[]~"
+    try:
+        # Unquote only the unreserved characters
+        # Then quote only illegal characters (do not quote reserved,
+        # unreserved, or '%')
+        return quote(unquote_unreserved(uri), safe=safe_with_percent)
+    except InvalidURL:
+        # We couldn't unquote the given URI, so let's try quoting it, but
+        # there may be unquoted '%'s in the URI. We need to make sure they're
+        # properly quoted so they do not cause issues elsewhere.
+        return quote(uri, safe=safe_without_percent)


Test patch:
diff --git a/tests/test_helpers.py b/tests/test_helpers.py
--- a/tests/test_helpers.py
+++ b/tests/test_helpers.py
@@ -133,3 +133,17 @@ def test_get_lower(self):
         self.assertEqual(atoms['{TEST}o'], '123')
         self.assertEqual(atoms['{UNKNOWN}o'], '-')
         self.assertEqual(atoms['{UNKNOWN}'], '-')
+
+
+class TestRequoting(unittest.TestCase):
+
+    def test_requote_uri_with_unquoted_percents(self):
+        # Ensure we handle unquoted percent signs in redirects.
+        bad_uri = 'http://example.com/fiz?buz=%ppicture'
+        quoted = 'http://example.com/fiz?buz=%25ppicture'
+        self.assertEqual(quoted, helpers.requote_uri(bad_uri))
+
+    def test_requote_uri_properly_requotes(self):
+        # Ensure requoting doesn't break expectations.
+        quoted = 'http://example.com/fiz?buz=%25ppicture'
+        self.assertEqual(quoted, helpers.requote_uri(quoted))