Repository: urllib3/urllib3

Issue:
Issue with Parsing URIs - Breaks Security Tools when testing for Path Traversal 
Hi all,

I have noticed an odd behaviour in requests module, which uses urllib3. I inspected the root cause via regression testing and I found that the root cause of the issue is a change that was introduced in urlib3 

```python
import requests
requests.get("http://127.0.0.1/../../../../doing/certain/check")
```
Which should typically send to the following path: `/../../../doing/certain/check`. Instead, it's requesting `/doing/certain/check` and modifying the specified input.

This breaks a lot of security tools that are performing fuzzing for path traversal vulnerabilities in web-applications.

The only solution for current tools is to revert to `urlib3==1.24.3`.

This may be related: https://github.com/urllib3/urllib3/commit/0aa3e24fcd75f1bb59ab159e9f8adb44055b2271#diff-26a37c674beb46c0ae8f77d76946e054


Can we work on fixing this issue?

 


Discussion:
Yeah I agree we should keep the previous behavior. My thoughts are that we should only do path normalization on redirects. Are you able to potentially provide a PR to fix this issue?
Hi @sethmlarson 

Thanks for your response. 

I have spent more time today morning to find the root cause. The normalization happens at `urlib3/util/url.py`. It was introduced in this commit https://github.com/urllib3/urllib3/commit/5b047b645f5f93900d5e2fc31230848c25eb1f5f#diff-c289049c6f3c9d7397378a8be7e700c6. 

I have made a PR that can be found here: https://github.com/urllib3/urllib3/pull/1792

Unfortunately making a specific use-case where to normalize URI when doing a redirect sounds difficult. I'm not familiar with urlib3 codebase, but this PR should be a fix for the base behaviour. 

Thanks again!

I'm sorry that urllib3 broke your workflow, and I would also like to apologize for [ruining your weekend](https://twitter.com/mazen160/status/1220738021626208256).

I'm not an Information Security Specialist like you are and am not the lead maintainer, but I'd like to step back for a moment. It seems to me that the fact that we were able to fix this vulnerability for all our users is actually a good thing! And I'm sure you'll appreciate that lowering our security in the name of security is quite ironic.

Are there other ways to check for this vulnerability? Maybe using a lower-level tool like [http.client](https://docs.python.org/3/library/http.client.html) would be more appropriate here?
Hi @pquentin !

Thanks for your response. It was fun debugging for this issue actually :)

This issue does not lower the security of urllib3 by any means. It can be a direct security issue if urllib3 is a server, but a client lib for sending HTTP requests should not be affected by how URLs are being normalized before sending with the `../../` case.

I'm currently writing a wrapper around the function responsible for this test in the framework, but a global fix for this issue would be nicer for everyone:)

Everyone is relying on urllib3 and requests module, as it's the expected behaviour; to send user-supplied requests as supplied, with minimal normalization if it's really needed. 

For example, filtering/blocking requests for CRLF user-supplied input makes sense. Blocking schemes that can cause further damage makes sense. Etc... However, normalization `../../` does not protect against an attack vector, and removing this normalization case is does not introduce one.

It would be great if anyone can suggest a possible attack vector in the `../../` case for urllib3.
Thanks for the friendly tone, @mazen160! It's appreciated as it makes a huge difference for maintainers and can only make the discussion more constructive.

I agree that the path traversal vulnerability is a server vulnerability, not a client vulnerability. There are some cases where as an attacker you won't control the client code, but can ultimately control the URL, so if the client normalizes the URL, then you can no longer exploit the vulnerability. It's not very likely, but defense in depth suggests that we should still do this.

Another argument here is that RFC 3986 says that clients should [remove dot segments](https://tools.ietf.org/html/rfc3986#section-6.2.2.3) and even provides a [reference implementation](https://tools.ietf.org/html/rfc3986#section-5.2.4). Chrome and curl do this, and I think Firefox does it too (but am not sure). However curl provides an escape hatch, which we can't do easily as a library. :/
Hi @pquentin! Thanks!

I agree with your approach, if there is a way to have an option like in curl, this will be excellent.

I have also found another project that were discussing the same issue here. They also provided a different solution in requests module.
- https://github.com/trustedsec/cve-2019-19781/blob/master/citrixmash.py#L140
- https://github.com/trustedsec/cve-2019-19781/issues/13

I wrote a wrapper around urlib.request to overcome the issue for these specific requests.
Feel free to choose the best decision to make from your side :)
Oh, setting the URL directly in a prepared request is a great find. It works because it bypasses the parse at the requests level and then at the urllib3 level (since requests uses connection pools directly, not pool managers).

It's equivalent to doing this with urllib3:

```
import urllib3

pool = urllib3.HTTPConnectionPool("localhost", 8000)
r = pool.urlopen("GET", "/../../../../doing/certain/check")
print(r.status)
```

(Can be tested using this server: `python3 -m http.server`.)

Since we have three possible workarounds, maybe we should close this and the associated pull request! @sethmlarson What do you think?
I think we should be good with those alternatives. Can we add a simple test case to ensure we preserve the non-normalizing behavior of using a ConnectionPool?
Might be worth documenting it as an escape hatch as well?
Sounds good to me. @mazen160 Is this something you'd be interested in working on? That is, testing that ConnectionPool does not normalize URLs, and add it to the documentation. I don't know how hard it would be to write the test, but we'd be happy to help

Resolving patch:
diff --git a/dummyserver/handlers.py b/dummyserver/handlers.py
--- a/dummyserver/handlers.py
+++ b/dummyserver/handlers.py
@@ -93,7 +93,7 @@ def _call_method(self):
         if not path.startswith("/"):
             path = urlsplit(path).path
 
-        target = path[1:].replace("/", "_")
+        target = path[1:].split("/", 1)[0]
         method = getattr(self, target, self.index)
 
         resp = method(req)
diff --git a/src/urllib3/connectionpool.py b/src/urllib3/connectionpool.py
--- a/src/urllib3/connectionpool.py
+++ b/src/urllib3/connectionpool.py
@@ -65,6 +65,11 @@ class ConnectionPool(object):
     """
     Base class for all connection pools, such as
     :class:`.HTTPConnectionPool` and :class:`.HTTPSConnectionPool`.
+
+    .. note::
+       ConnectionPool.urlopen() does not normalize or percent-encode target URIs
+       which is useful if your target server doesn't support percent-encoded
+       target URIs.
     """
 
     scheme = None


Test patch:
diff --git a/test/with_dummyserver/test_connectionpool.py b/test/with_dummyserver/test_connectionpool.py
--- a/test/with_dummyserver/test_connectionpool.py
+++ b/test/with_dummyserver/test_connectionpool.py
@@ -787,6 +787,12 @@ def test_mixed_case_hostname(self):
             response = pool.request("GET", "http://LoCaLhOsT:%d/" % self.port)
             assert response.status == 200
 
+    def test_preserves_path_dot_segments(self):
+        """ ConnectionPool preserves dot segments in the URI """
+        with HTTPConnectionPool(self.host, self.port) as pool:
+            response = pool.request("GET", "/echo_uri/seg0/../seg2")
+            assert response.data == b"/echo_uri/seg0/../seg2"
+
 
 class TestRetry(HTTPDummyServerTestCase):
     def test_max_retry(self):