Repository: urllib3/urllib3

Issue:
Add tilde to ZONE_ID_CHARS
Fixes #1683.

Recently merged #1673 ensured that URLs are percent-encoded.
This is a behavior change in that URLs with tilde characters in
them would not previously have been percent-encoded, but are now.

RFC 3986 says[1]:

    For consistency, percent-encoded octets in the ranges of ALPHA
    (%41-%5A and %61-%7A), DIGIT (%30-%39), hyphen (%2D), period (%2E),
    underscore (%5F), or tilde (%7E) should not be created by URI
    producers and, when found in a URI, should be decoded to their
    corresponding unreserved characters by URI normalizers.

This suggests that urllib3 should not escape tilde characters
in URLs.  The RFC describes tilde as an "unreserved" character.
Among the character classes at the top of url.py, the closest
match to the "unreserved" set seems to be ZONE_ID_CHARS.  RFC6874
says[2]:

    A <zone_id> SHOULD contain only ASCII characters classified as
    "unreserved" for use in URIs [RFC3986].

Which suggests that it should be safe to add to that set.

[1] https://tools.ietf.org/html/rfc3986#section-2.3
[2] https://tools.ietf.org/html/rfc6874#section-2


Discussion:
# [Codecov](https://codecov.io/gh/urllib3/urllib3/pull/1684?src=pr&el=h1) Report
> Merging [#1684](https://codecov.io/gh/urllib3/urllib3/pull/1684?src=pr&el=desc) into [master](https://codecov.io/gh/urllib3/urllib3/commit/7e856c04723036934fe314c63701466e4f42d2ee?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/urllib3/urllib3/pull/1684/graphs/tree.svg?width=650&token=UYKHB2uXDn&height=150&src=pr)](https://codecov.io/gh/urllib3/urllib3/pull/1684?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1684   +/-   ##
=======================================
  Coverage   99.45%   99.45%           
=======================================
  Files          22       22           
  Lines        2005     2005           
=======================================
  Hits         1994     1994           
  Misses         11       11
```


| [Impacted Files](https://codecov.io/gh/urllib3/urllib3/pull/1684?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/urllib3/util/url.py](https://codecov.io/gh/urllib3/urllib3/pull/1684/diff?src=pr&el=tree#diff-c3JjL3VybGxpYjMvdXRpbC91cmwucHk=) | `98.98% <ø> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/urllib3/urllib3/pull/1684?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/urllib3/urllib3/pull/1684?src=pr&el=footer). Last update [7e856c0...4f65f3a](https://codecov.io/gh/urllib3/urllib3/pull/1684?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Resolving patch:
diff --git a/src/urllib3/util/url.py b/src/urllib3/util/url.py
--- a/src/urllib3/util/url.py
+++ b/src/urllib3/util/url.py
@@ -50,7 +50,7 @@
     "(?:(?:%(hex)s:){0,6}%(hex)s)?::",
 ]
 
-UNRESERVED_PAT = r"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789._!\-"
+UNRESERVED_PAT = r"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789._!\-~"
 IPV6_PAT = "(?:" + "|".join([x % _subs for x in _variations]) + ")"
 ZONE_ID_PAT = "(?:%25|%)(?:[" + UNRESERVED_PAT + "]|%[a-fA-F0-9]{2})+"
 IPV6_ADDRZ_PAT = r"\[" + IPV6_PAT + r"(?:" + ZONE_ID_PAT + r")?\]"
@@ -63,17 +63,18 @@
 BRACELESS_IPV6_ADDRZ_RE = re.compile("^" + IPV6_ADDRZ_PAT[2:-2] + "$")
 ZONE_ID_RE = re.compile("(" + ZONE_ID_PAT + r")\]$")
 
-SUBAUTHORITY_PAT = (u"^(?:(.*)@)?" u"(%s|%s|%s)" u"(?::([0-9]{0,5}))?$") % (
+SUBAUTHORITY_PAT = (u"^(?:(.*)@)?(%s|%s|%s)(?::([0-9]{0,5}))?$") % (
     REG_NAME_PAT,
     IPV4_PAT,
     IPV6_ADDRZ_PAT,
 )
 SUBAUTHORITY_RE = re.compile(SUBAUTHORITY_PAT, re.UNICODE | re.DOTALL)
 
-ZONE_ID_CHARS = set(
-    "ABCDEFGHIJKLMNOPQRSTUVWXYZ" "abcdefghijklmnopqrstuvwxyz" "0123456789._!-"
+UNRESERVED_CHARS = set(
+    "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789._-~"
 )
-USERINFO_CHARS = ZONE_ID_CHARS | set("$&'()*+,;=:")
+SUB_DELIM_CHARS = set("!$&'()*+,;=")
+USERINFO_CHARS = UNRESERVED_CHARS | SUB_DELIM_CHARS | {":"}
 PATH_CHARS = USERINFO_CHARS | {"@", "/"}
 QUERY_CHARS = FRAGMENT_CHARS = PATH_CHARS | {"?"}
 
@@ -290,7 +291,7 @@ def _normalize_host(host, scheme):
                         zone_id = zone_id[3:]
                     else:
                         zone_id = zone_id[1:]
-                    zone_id = "%" + _encode_invalid_chars(zone_id, ZONE_ID_CHARS)
+                    zone_id = "%" + _encode_invalid_chars(zone_id, UNRESERVED_CHARS)
                     return host[:start].lower() + zone_id + host[end:]
                 else:
                     return host.lower()


Test patch:
diff --git a/test/test_util.py b/test/test_util.py
--- a/test/test_util.py
+++ b/test/test_util.py
@@ -170,6 +170,10 @@ def test_invalid_url(self, url):
         "url, expected_normalized_url",
         [
             ("HTTP://GOOGLE.COM/MAIL/", "http://google.com/MAIL/"),
+            (
+                "http://user@domain.com:password@example.com/~tilde@?@",
+                "http://user%40domain.com:password@example.com/~tilde@?@",
+            ),
             (
                 "HTTP://JeremyCline:Hunter2@Example.com:8080/",
                 "http://JeremyCline:Hunter2@example.com:8080/",