Repository: encode/httpx

Issue:
Square brackets no longer work
A GET request such as `/?arr[]=1&arr[]=2` is perfectly legal and it means that you are trying to send an array of values to the endpoint.

Unluckily after 0.23.3 it stopped working because requests are sent as `/?arr%5B%5D=1&arr%5B%5D=2`






Discussion:
hrm... my reading of RFC 3986 is that square brackets *do* need escaping in the path component.

See https://www.rfc-editor.org/rfc/rfc3986#section-3.3

```
pchar         = unreserved / pct-encoded / sub-delims / ":" / "@"
```

And...

https://www.rfc-editor.org/rfc/rfc3986#section-2.2

```
      reserved    = gen-delims / sub-delims

      gen-delims  = ":" / "/" / "?" / "#" / "[" / "]" / "@"

      sub-delims  = "!" / "$" / "&" / "'" / "(" / ")"
                  / "*" / "+" / "," / ";" / "="
```

The condensed https://www.rfc-editor.org/rfc/rfc3986#appendix-A is helpful there.

Am I reading this wrong?

However, I can see that chrome does *not* percent-escape square brackets in the path. Perhaps the browser behaviour differs from the RFC spec?


You are actually right, this is poorly documented and looks like sending non escaped square brackets is a standard de facto. 

The ugly solution would be to add a `strict` parameter, it's definitely breaking something that used to work in the past :) 



If browser behaviour doesn't match the spec here, then I'm okay with `httpx` switching to that.

A pull request adding them to the unescaped sequences, along with a code comment would be okay.

Resolving patch:
diff --git a/httpx/_urlparse.py b/httpx/_urlparse.py
--- a/httpx/_urlparse.py
+++ b/httpx/_urlparse.py
@@ -253,12 +253,19 @@ def urlparse(url: str = "", **kwargs: typing.Optional[str]) -> ParseResult:
     if has_authority:
         path = normalize_path(path)
 
-    parsed_path: str = quote(path, safe=SUB_DELIMS + ":@/")
+    # The GEN_DELIMS set is... : / ? # [ ] @
+    # These do not need to be percent-quoted unless they serve as delimiters for the
+    # specific component.
+
+    # For 'path' we need to drop ? and # from the GEN_DELIMS set.
+    parsed_path: str = quote(path, safe=SUB_DELIMS + ":/[]@")
+    # For 'query' we need to drop '#' from the GEN_DELIMS set.
     parsed_query: typing.Optional[str] = (
-        None if query is None else quote(query, safe=SUB_DELIMS + "/?")
+        None if query is None else quote(query, safe=SUB_DELIMS + ":/?[]@")
     )
+    # For 'fragment' we can include all of the GEN_DELIMS set.
     parsed_fragment: typing.Optional[str] = (
-        None if fragment is None else quote(fragment, safe=SUB_DELIMS + "/?")
+        None if fragment is None else quote(fragment, safe=SUB_DELIMS + ":/?#[]@")
     )
 
     # The parsed ASCII bytestrings are our canonical form.


Test patch:
diff --git a/tests/test_urlparse.py b/tests/test_urlparse.py
--- a/tests/test_urlparse.py
+++ b/tests/test_urlparse.py
@@ -247,3 +247,33 @@ def test_copy_with():
 
     url = url.copy_with(path="/abc")
     assert str(url) == "http://example.com/abc"
+
+
+# Tests for percent encoding across path, query, and fragement...
+
+
+def test_path_percent_encoding():
+    # Test percent encoding for SUB_DELIMS ALPHA NUM and allowable GEN_DELIMS
+    url = httpx.URL("https://example.com/!$&'()*+,;= abc ABC 123 :/[]@")
+    assert url.raw_path == b"/!$&'()*+,;=%20abc%20ABC%20123%20:/[]@"
+    assert url.path == "/!$&'()*+,;= abc ABC 123 :/[]@"
+    assert url.query == b""
+    assert url.fragment == ""
+
+
+def test_query_percent_encoding():
+    # Test percent encoding for SUB_DELIMS ALPHA NUM and allowable GEN_DELIMS
+    url = httpx.URL("https://example.com/?!$&'()*+,;= abc ABC 123 :/[]@" + "?")
+    assert url.raw_path == b"/?!$&'()*+,;=%20abc%20ABC%20123%20:/[]@?"
+    assert url.path == "/"
+    assert url.query == b"!$&'()*+,;=%20abc%20ABC%20123%20:/[]@?"
+    assert url.fragment == ""
+
+
+def test_fragment_percent_encoding():
+    # Test percent encoding for SUB_DELIMS ALPHA NUM and allowable GEN_DELIMS
+    url = httpx.URL("https://example.com/#!$&'()*+,;= abc ABC 123 :/[]@" + "?#")
+    assert url.raw_path == b"/"
+    assert url.path == "/"
+    assert url.query == b""
+    assert url.fragment == "!$&'()*+,;= abc ABC 123 :/[]@?#"