Repository: pallets/werkzeug

Issue:
ProxyMiddleware encodes characters allowed in path
We have a dollar sign in URL and proxy middleware converts it to `%24`. Unfortunately, for backend server url with `$` and url with `%24` are two different urls. It could be fixed by extending safe characters in middleware as it done in [url_fix](https://github.com/pallets/werkzeug/blob/9efe8c00dcb2b6fc086961ba304729db01912652/src/werkzeug/urls.py#L691) function, but it seems to me that the right way is to make the middleware to use `REQUEST_URI`/`RAW_URI` when possible. This way also can fix middleware for cases like the one described in https://github.com/pallets/werkzeug/issues/766.

I can send PR, but since it will require some efforts on making `remove_prefix` option working correctly and to ensure backward compatibility for public method `proxy_to`, I'd like to discuss this solution first. If it can be accepted, then I proceed.


Discussion:
Sounds like `$` should be in the `always_safe` set of characters. If you can find the most up-to-date RFC describing URL safe characters, I'd be happy to accept a PR that updates the code accordingly. However, you need to find that reference first.
https://url.spec.whatwg.org/#percent-encoded-bytes

C0 Controls are characters <= 0x1f.
`The C0 control percent-encode set are the C0 controls and all code points greater than U+007E (~).`
`The query percent-encode set is the C0 control percent-encode set and U+0020 SPACE, U+0022 ("), U+0023 (#), U+003C (<), and U+003E (>).`
``The path percent-encode set is the query percent-encode set and U+003F (?), U+0060 (`), U+007B ({), and U+007D (}).``

So list of characters requiring percent-encoding in path segment: ``"#<>?`{}`` + control characters with code less than code of space and space itself.

What about REQUEST_URI, wouldn't it be better to keep resulting URL as close as possible to the original one?
I don't know what `REQUEST_URI` you're referring to, that doesn't appear in the proxy middleware module.
Sorry, I mean, what if instead of [PATH_INFO](https://github.com/pallets/werkzeug/blob/9efe8c00dcb2b6fc086961ba304729db01912652/src/werkzeug/middleware/http_proxy.py#L222) use REQUEST_URI variable, so it would not require to reconstruct url and that url would contain all the same encoded characters as requested by user. If user requested `/proxy/$` then proxy will request `/$`, if user requested `/proxy/%24` proxy will send request to `/%24`.
`REQUEST_URI` is not standardized in the WSGI environ. It is not generally possible to guarantee it is available or accurate, or even named consistently (some servers set `RAW_URI`).
According to [RFC3986](https://www.rfc-editor.org/rfc/rfc3986#section-2.2) `sub-delims` ``$!&'()*+,;=`` are valid characters in [path](https://www.rfc-editor.org/rfc/rfc3986#section-3.3), [query](https://www.rfc-editor.org/rfc/rfc3986#section-3.4) (included in `pchar` set), [fragment](https://www.rfc-editor.org/rfc/rfc3986#section-3.5), [host](https://www.rfc-editor.org/rfc/rfc3986#section-3.2.2) and [user info](https://www.rfc-editor.org/rfc/rfc3986#section-3.2.1) and should not be escaped. WHATWG document confirms that, see https://url.spec.whatwg.org/#host-miscellaneous for hosts limitation.

Both RFC and WHATWG allows these characters in host, but current implementation of IPv* addresses and domain names don't utilize them.

Also [tested](https://jsdom.github.io/whatwg-url/#url=aHR0cHM6Ly9leGFtcGxlLmNvbS9wYXRoJCEmJygpKissOz0/cXVlcnkkISYnKCkqKyw7PSNmcmFnbWVudCQhJicoKSorLDs9&base=YWJvdXQ6Ymxhbms=) all these delimiters with Live URL Viewer tool, it only fails on [host](https://jsdom.github.io/whatwg-url/#url=aHR0cHM6Ly91c2VyJCEmJygpKissOz06cGFzcyQhJicoKSorLDs9QGhvc3QkISYnKCkqKyw7PS8=&base=YWJvdXQ6Ymxhbms=) for my browser implementation, but the same happens with `~` already included to `always_safe`.

I suggest to include all ``$!&'()*+,;=`` characters in `always_safe`. We can ignore issue with host, as `url_quote` never used to quote host name and `%` is invalid character in host in last WHATWG document anyway.



Happy to review a PR. 

Resolving patch:
diff --git a/src/werkzeug/urls.py b/src/werkzeug/urls.py
--- a/src/werkzeug/urls.py
+++ b/src/werkzeug/urls.py
@@ -27,6 +27,7 @@
         b"ABCDEFGHIJKLMNOPQRSTUVWXYZ"
         b"0123456789"
         b"-._~"
+        b"$!'()*+,;"  # RFC3986 sub-delims set, not including query string delimiters &=
     )
 )
 


Test patch:
diff --git a/tests/middleware/test_http_proxy.py b/tests/middleware/test_http_proxy.py
--- a/tests/middleware/test_http_proxy.py
+++ b/tests/middleware/test_http_proxy.py
@@ -45,3 +45,7 @@ def test_http_proxy(standard_app):
     assert "HTTP_X_SPECIAL" not in r.json
     assert r.json["HTTP_HOST"] == "127.0.0.1"
     assert r.json["PATH_INFO"] == "/autohost/aha"
+
+    # test if characters allowed in URL are not encoded by proxy
+    r = client.get("/autohost/$")
+    assert r.json["REQUEST_URI"] == "/autohost/$"