full URL sent instead of path when using https proxy

When using the requests library through a web proxy, requests includes a full URL in the request line instead of just the path for all https requests.  Http requests send just the path, as expected. Https requests send the full URL in place of the path.  This can cause problems with certain proxies (for example, the BURP proxy (http://portswigger.net/burp/download.html) will not properly process these requests unless the special "transparent" proxy mode is enabled).

The following script demonstrates his problem:

```
#!/usr/bin/env python
"""
Requests library will send full URL as the request path
when sending HTTPS requests to a proxy.. WHY?

Run this script through a proxy listening on the local host,
such as BURP, which records all traffic.  Look at the proxy
logs to see the request lines.  Notice that this will send
a full URL in the request line.
"""

import requests

proxies = { 
   'http': 'http://127.0.0.1:8080/',
   'https': 'https://127.0.0.1:8080/' 
}

# sends full URL in the request line:
r = requests.get("https://www.yahoo.com/the/path/goes/here", proxies=proxies, allow_redirects=False)

# sends full just the path in the request line:
r = requests.get("http://www.yahoo.com/the/path/goes/here", proxies=proxies, allow_redirects=False)
```


Hmm, I'm honestly not sure. My gut is saying that this is because having a HTTPS scheme in a proxy URL just doesn't really work.


So I think this might also have something to do with the `CONNECT` verb which we use when connecting to an `https` proxy. I know that for `CONNECT` you have to pass the authority portion of the request URI (e.g., `www.yahoo.com:443`) but I'm not sure if the RFC (7231 or 7232 now) says anything about including the path. I might look that up over lunch


Compare this to the behavior of all other http request libraries (such as urllib) which do not send a full path for https proxies.

Tim Newsham | tim@isecpartners.commailto:tim@isecpartners.com | PGP: B415 550D BEE9 07DB B4C9  F96C 8EFE CB2F 402D 3DF0

From: Cory Benfield [mailto:notifications@github.com]
Sent: Tuesday, September 16, 2014 8:17 PM
To: kennethreitz/requests
Cc: Tim Newsham
Subject: Re: [requests] full URL sent instead of path when using https proxy (#2229)

Hmm, I'm honestly not sure. My gut is saying that this is because having a HTTPS scheme in a proxy URL just doesn't really work.

—
Reply to this email directly or view it on GitHubhttps://github.com/kennethreitz/requests/issues/2229#issuecomment-55853284.


To be fair, `urllib` is broken in many ways, so frankly comparing requests' behaviour to urllib's is not exactly a comparison that holds weight for me. If instead, you have a comparison to `cURL` that would hold more weight. Further if you can find the section in the RFC that declares how this should work, that'd be best.


Here's what I was able to come up with.

According to [RFC 7230](https://tools.ietf.org/html/rfc7230#section-5.3.2):

> When making a request to a proxy, other than a CONNECT or server-wide OPTIONS request (as detailed below), a client MUST send the target URI in absolute-form as the request-target.... An example absolute-form of request-line would be:
> GET http://www.example.org/pub/WWW/TheProject.html HTTP/1.1

But the `HTTPAdapter.proxy_manager_for` method uses urllib3's `ProxyManager`, which uses CONNECT for https requests. [A paragraph down in RFC 7230](https://tools.ietf.org/html/rfc7230#section-5.3.3):

> When making a CONNECT request to establish a tunnel through one or more proxies, a client MUST send only the target URI's authority component (excluding any userinfo and its "@" delimiter) as the request-target.  For example,
> CONNECT www.example.com:80 HTTP/1.1

Therefore, it looks to me like https requests should always only pass the authority-form of the request target (i.e. the path, not the full url). Then @newsham is seeing a bug in the https requests are handled. Am I missing anything?

However, I wasn't able to reproduce the problem. On my own installation of BURP, the demonstration script created two requests, both of which were logged by BURP as only including the path (correct behavior). Furthermore the method was listed as GET for the https request, but I was expecting CONNECT. Could this be my unfamiliarity with BURP?


@benjaminran Nope, it's a corner case in how requests handles proxies.

If you pass a proxy whose proxy url scheme is 'https://', we don't use CONNECT to tunnel through it, because what's the point? Instead, we connect directly to it over TLS and then treat it like a plaintext proxy, which explains the behaviour you're seeing.


I thought I tracked down the code that handles that when I found the urllib3 ProxyManager in adapters.py:245. Where's the section that specifies to use TLS?

The first quotation (5.3.2) from RFC 7230 suggests that the behavior newsham is seeing is actually correct--most https requests should be sent with the target as a full url. But I still can't replicate the behavior. I'm running newsham's demonstration script into BURP and both requests have origin-form targets (paths, not urls). Is anyone else able to replicate this?


I can no longer reproduce this issue with my original reproduction script, either.


Closed as unreproducible. Please reopen if a new repro appears.