Repository: urllib3/urllib3

Issue:
parse_url no longer accepts [] in the URL
With urllib3 1.25.1:

```
>>> parse_url('https://example.com/somevar[]')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/testrunner/testvenv/lib/python2.7/site-packages/urllib3/util/url.py", line 212, in parse_url
    six.raise_from(LocationParseError(url), None)
  File "/home/testrunner/testvenv/lib/python2.7/site-packages/urllib3/packages/six.py", line 718, in raise_from
    raise value
urllib3.exceptions.LocationParseError: Failed to parse: https://example.com/somevar[]
```

Using 1.24.2:

```
>>> parse_url('https://example.com/somevar[]')
Url(scheme='https', auth=None, host='example.com', port=None, path='/somevar[]', query=None, fragment=None)
```

```
>>> parse_url('https://example.com/?somevar["whatever"]')
Url(scheme='https', auth=None, host='example.com', port=None, path='/', query='somevar["whatever"]', fragment=None)
```

This is used by various bits of software (ElasticSearch for instance):

```
>>> parse_url('http://example.com/api/search/?grouping=datasets&q=dontcare&projection=["variables.nonexistingfield"]')
Url(scheme='http', auth=None, host='example.com', port=None, path='/api/search/', query='grouping=datasets&q=dontcare&projection=["variables.nonexistingfield"]', fragment=None)
```


Discussion:
Per [RFC 3986 Appendix A](https://tools.ietf.org/html/rfc3986) the only valid characters in the path component are `unreserved`, `pct-encoded`, `sub-delims`, `:`, and `@` which doesn't include `[` or `]`. Can you see if percent-encoding the square braces works as you expect? Maybe we need to do this as a part of normalization?
Okay, this time not in the path:

```
>>> parse_url('http://example.com/api/search/?grouping=datasets&q=dontcare&projection=["variables.nonexistingfield"]')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/testrunner/testvenv/lib/python2.7/site-packages/urllib3/util/url.py", line 212, in parse_url
    six.raise_from(LocationParseError(url), None)
  File "/home/testrunner/testvenv/lib/python2.7/site-packages/urllib3/packages/six.py", line 718, in raise_from
    raise value
urllib3.exceptions.LocationParseError: Failed to parse: http://example.com/api/search/?grouping=datasets&q=dontcare&projection=["variables.nonexistingfield"]
```

You'll find plenty of PHP sites out there that use:

```
>>> parse_url('http://www.example.com/foo.php?bar[]=1&bar[]=2&bar[]=3')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/testrunner/testvenv/lib/python2.7/site-packages/urllib3/util/url.py", line 212, in parse_url
    six.raise_from(LocationParseError(url), None)
  File "/home/testrunner/testvenv/lib/python2.7/site-packages/urllib3/packages/six.py", line 718, in raise_from
    raise value
urllib3.exceptions.LocationParseError: Failed to parse: http://www.example.com/foo.php?bar[]=1&bar[]=2&bar[]=3
```

This creates an array in `$_GET` in PHP for instance.

I understand that this may not be valid as a URL to enter directly, but it is data we have to parse and `parse_url` usually works wonders for that.

How would you recommend I take a user provided string, pull apart the pieces to URL encode the brackets and the like, because `urllib.quote` on the whole thing will create this mess:

```
>>> urllib.quote('http://www.example.com/foo.php?bar[]=1&bar[]=2&bar[]=3')
'http%3A//www.example.com/foo.php%3Fbar%5B%5D%3D1%26bar%5B%5D%3D2%26bar%5B%5D%3D3'
>>> parse_url(urllib.quote('http://www.example.com/foo.php?bar[]=1&bar[]=2&bar[]=3'))
Url(scheme=None, auth=None, host='http%3a', port=None, path='//www.example.com/foo.php%3Fbar%5B%5D%3D1%26bar%5B%5D%3D2%26bar%5B%5D%3D3', query=None, fragment=None)
```
If you're looking for an alternative to `parse_url` you can check out the library we're using for URL parsing ([`rfc3986`](https://github.com/python-hyper/rfc3986)) and then not do any validation on the path or query components? Then you can encode the brackets yourself using potentially `urllib.quote()` or something like it.
I think we should consider percent-encoding invalid characters in the path, query, and fragment components automatically as a part of normalization though.  Thoughts @urllib3/maintainers?
Do we automatically percent encode any other stuff?

On Thu, Apr 25, 2019, 10:44 AM Seth Michael Larson <notifications@github.com>
wrote:

> I think we should consider percent-encoding invalid characters in the
> path, query, and fragment components automatically as a part of
> normalization though. Thoughts @urllib3/maintainers
> <https://github.com/orgs/urllib3/teams/maintainers>?
>
> —
> You are receiving this because you are on a team that was mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/urllib3/urllib3/issues/1585#issuecomment-486770925>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAB5I42ZPAN7D7WOUAPQSFLPSHUWRANCNFSM4HIPAGXQ>
> .
>

No we don't currently. But previously we just passed through whatever the user gave us in the path and query components into the target. We want to do validation at least for control characters at a minimum but I don't know if anything else can hurt us here.

So I guess another option is to not normalize and validate only for the bare minimum otherwise pass-through.
I would like to point out that we found the issue in urllib3 because it was accidentally upgraded by pip and requests just passes the URL through unmodified.

So we would potentially have to intercept the URL before passing it to requests (as well as deal with any `Location` headers/redirects ourselves) just to modify it before passing it to requests which passes it to `urllib3`.
Hrm. I'm okay with being a little lax on path and query validation.
Ultimately it's up to the server to decide if a request is valid or not.

On Thu, Apr 25, 2019, 10:52 AM Seth Michael Larson <notifications@github.com>
wrote:

> No we don't currently. But previously we just passed through whatever the
> user gave us in the path and query components into the target. We want to
> do validation at least for control characters at a minimum but I don't know
> if anything else can hurt us here.
>
> —
> You are receiving this because you are on a team that was mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/urllib3/urllib3/issues/1585#issuecomment-486773960>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAB5I466G2X3IJSBHQC66OTPSHVWHANCNFSM4HIPAGXQ>
> .
>

@theacodes Actually scratch that we totally do inside of `IRIReference.from_string()`. The path, query and fragment components all receive some percent encoding just not for `[` or `]`.

Resolving patch:
diff --git a/src/urllib3/util/url.py b/src/urllib3/util/url.py
--- a/src/urllib3/util/url.py
+++ b/src/urllib3/util/url.py
@@ -6,6 +6,7 @@
 from ..packages import six, rfc3986
 from ..packages.rfc3986.exceptions import RFC3986Exception, ValidationError
 from ..packages.rfc3986.validators import Validator
+from ..packages.rfc3986 import abnf_regexp, normalizers, compat, misc
 
 
 url_attrs = ['scheme', 'auth', 'host', 'port', 'path', 'query', 'fragment']
@@ -17,6 +18,9 @@
 # Regex for detecting URLs with schemes. RFC 3986 Section 3.1
 SCHEME_REGEX = re.compile(r"^(?:[a-zA-Z][a-zA-Z0-9+\-]*:|/)")
 
+PATH_CHARS = abnf_regexp.UNRESERVED_CHARS_SET | abnf_regexp.SUB_DELIMITERS_SET | {':', '@', '/'}
+QUERY_CHARS = FRAGMENT_CHARS = PATH_CHARS | {'?'}
+
 
 class Url(namedtuple('Url', url_attrs)):
     """
@@ -136,6 +140,37 @@ def split_first(s, delims):
     return s[:min_idx], s[min_idx + 1:], min_delim
 
 
+def _encode_invalid_chars(component, allowed_chars, encoding='utf-8'):
+    """Percent-encodes a URI component without reapplying
+    onto an already percent-encoded component. Based on
+    rfc3986.normalizers.encode_component()
+    """
+    if component is None:
+        return component
+
+    # Try to see if the component we're encoding is already percent-encoded
+    # so we can skip all '%' characters but still encode all others.
+    percent_encodings = len(normalizers.PERCENT_MATCHER.findall(
+                            compat.to_str(component, encoding)))
+
+    uri_bytes = component.encode('utf-8', 'surrogatepass')
+    is_percent_encoded = percent_encodings == uri_bytes.count(b'%')
+
+    encoded_component = bytearray()
+
+    for i in range(0, len(uri_bytes)):
+        # Will return a single character bytestring on both Python 2 & 3
+        byte = uri_bytes[i:i+1]
+        byte_ord = ord(byte)
+        if ((is_percent_encoded and byte == b'%')
+                or (byte_ord < 128 and byte.decode() in allowed_chars)):
+            encoded_component.extend(byte)
+            continue
+        encoded_component.extend('%{0:02x}'.format(byte_ord).encode().upper())
+
+    return encoded_component.decode(encoding)
+
+
 def parse_url(url):
     """
     Given a url, return a parsed :class:`.Url` namedtuple. Best-effort is
@@ -160,8 +195,6 @@ def parse_url(url):
         return Url()
 
     is_string = not isinstance(url, six.binary_type)
-    if not is_string:
-        url = url.decode("utf-8")
 
     # RFC 3986 doesn't like URLs that have a host but don't start
     # with a scheme and we support URLs like that so we need to
@@ -171,11 +204,6 @@ def parse_url(url):
     if not SCHEME_REGEX.search(url):
         url = "//" + url
 
-    try:
-        iri_ref = rfc3986.IRIReference.from_string(url, encoding="utf-8")
-    except (ValueError, RFC3986Exception):
-        six.raise_from(LocationParseError(url), None)
-
     def idna_encode(name):
         if name and any([ord(x) > 128 for x in name]):
             try:
@@ -188,8 +216,18 @@ def idna_encode(name):
                 raise LocationParseError(u"Name '%s' is not a valid IDNA label" % name)
         return name
 
-    has_authority = iri_ref.authority is not None
-    uri_ref = iri_ref.encode(idna_encoder=idna_encode)
+    try:
+        split_iri = misc.IRI_MATCHER.match(compat.to_str(url)).groupdict()
+        iri_ref = rfc3986.IRIReference(
+            split_iri['scheme'], split_iri['authority'],
+            _encode_invalid_chars(split_iri['path'], PATH_CHARS),
+            _encode_invalid_chars(split_iri['query'], QUERY_CHARS),
+            _encode_invalid_chars(split_iri['fragment'], FRAGMENT_CHARS)
+        )
+        has_authority = iri_ref.authority is not None
+        uri_ref = iri_ref.encode(idna_encoder=idna_encode)
+    except (ValueError, RFC3986Exception):
+        return six.raise_from(LocationParseError(url), None)
 
     # rfc3986 strips the authority if it's invalid
     if has_authority and uri_ref.authority is None:
@@ -209,7 +247,7 @@ def idna_encode(name):
             *validator.COMPONENT_NAMES
         ).validate(uri_ref)
     except ValidationError:
-        six.raise_from(LocationParseError(url), None)
+        return six.raise_from(LocationParseError(url), None)
 
     # For the sake of backwards compatibility we put empty
     # string values for path if there are any defined values


Test patch:
diff --git a/test/test_util.py b/test/test_util.py
--- a/test/test_util.py
+++ b/test/test_util.py
@@ -135,8 +135,15 @@ def test_invalid_host(self, location):
         'http://user\\@google.com',
         'http://google\\.com',
         'user\\@google.com',
-        'http://google.com#fragment#',
         'http://user@user@google.com/',
+
+        # Invalid IDNA labels
+        u'http://\uD7FF.com',
+        u'http://❤️',
+
+        # Unicode surrogates
+        u'http://\uD800.com',
+        u'http://\uDC00.com',
     ])
     def test_invalid_url(self, url):
         with pytest.raises(LocationParseError):
@@ -149,6 +156,15 @@ def test_invalid_url(self, url):
         ('HTTPS://Example.Com/?Key=Value', 'https://example.com/?Key=Value'),
         ('Https://Example.Com/#Fragment', 'https://example.com/#Fragment'),
         ('[::Ff%etH0%Ff]/%ab%Af', '[::ff%25etH0%Ff]/%AB%AF'),
+
+        # Invalid characters for the query/fragment getting encoded
+        ('http://google.com/p[]?parameter[]=\"hello\"#fragment#',
+         'http://google.com/p%5B%5D?parameter%5B%5D=%22hello%22#fragment%23'),
+
+        # Percent encoding isn't applied twice despite '%' being invalid
+        # but the percent encoding is still normalized.
+        ('http://google.com/p%5B%5d?parameter%5b%5D=%22hello%22#fragment%23',
+         'http://google.com/p%5B%5D?parameter%5B%5D=%22hello%22#fragment%23')
     ])
     def test_parse_url_normalization(self, url, expected_normalized_url):
         """Assert parse_url normalizes the scheme/host, and only the scheme/host"""
@@ -214,7 +230,14 @@ def test_parse_url_normalization(self, url, expected_normalized_url):
 
         # Uppercase IRI
         (u'http://Königsgäßchen.de/straße',
-         Url('http', host='xn--knigsgchen-b4a3dun.de', path='/stra%C3%9Fe'))
+         Url('http', host='xn--knigsgchen-b4a3dun.de', path='/stra%C3%9Fe')),
+
+        # Unicode Surrogates
+        (u'http://google.com/\uD800', Url('http', host='google.com', path='%ED%A0%80')),
+        (u'http://google.com?q=\uDC00',
+         Url('http', host='google.com', path='', query='q=%ED%B0%80')),
+        (u'http://google.com#\uDC00',
+         Url('http', host='google.com', path='', fragment='%ED%B0%80')),
     ]
 
     @pytest.mark.parametrize(