Incorrect Request When URL Contains Uppercase Letters in "HTTPS" Protocol and When Using Proxies

Hi there,

First off, Requests is awesome. Thank you for the great library!

I'm writing a script that places a series of requests and manually follows some redirects, and one of the redirects is to a URL that starts with "HTTPS" (that is, the protocol is uppercase, like HTTPS://www.example.com instead of https://www.example.com). It would seem that Requests generates the wrong request when the following three conditions are met:
1. The protocol is HTTPS.
2. The protocol contains one or more uppercase letters. (e.g. "HTTPS", "httpS", "hTTps", "Https", etc.)
3. Proxies are being used.

I wrote a test script that uses a Session to GET www.example.com with variations in case for both http and https protocols, and here are the results:

| proto | proxies | result |
| --- | --- | --- |
| http | no | [OK] |
| htTp | no | [OK] |
| HTTP | no | [OK] |
| https | no | [OK] |
| HTTPS | no | [OK] |
| htTps | no | [OK] |
| httpS | no | [OK] |
| Https | no | [OK] |
| http | YES | [OK] |
| htTp | YES | [OK] |
| HTTP | YES | [OK] |
| https | YES | [OK] |
| HTTPS | YES | [ERROR] |
| htTps | YES | [ERROR] |
| httpS | YES | [ERROR] |
| Https | YES | [ERROR] |

In the ERROR case, my proxy (Burp) was generating the following error:
**Error**
Invalid client request received: First line of request did not contain an absolute URL - try enabling invisible proxy support.
GET / HTTP/1.1
Host: localhost:8080
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: _/_
User-Agent: python-requests/2.9.1

The issue can be replicated with the following couple lines of code:
`import requests`
`requests.request('GET', 'HTTPS://www.example.com/test', proxies=dict(http='localhost:8080', https='localhost:8080'))`

What happens will depend on your proxy. With Burp, when errors are given back as valid HTTP responses, the above will result in a 200 OK response object, and when you inspect the content of the response, you'll see Burp's error. However, if you suppress Burp errors, then Burp returns an empty response, causing requests to raise a ConnectionError exception:
requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine("''",))

Thank you for your time!

Best Regards,
-Justin


Hey Justin! Good catch, this is definitely a bug.

The problem is that urllib3 expects only lowercase schemes in its proxy manager code. Requests could lowercase the scheme to urllib3 itself, but it seems like it'd be better to just make urllib3 tolerant of this kind of error.


Given that this problem is pretty pervasive in urllib3's code, I'm going to close this issue in favour of shazow/urllib3#833. Let's improve both projects at once.

Thanks for the report @correcthorsebatterystaple-, please feel free to subscribe to the issue linked above.


Hi @Lukasa, thanks for the quick response!

Is the urllib3 issue specific to https? Because it seems to work with mixed case protocol when it's http, and only has issues with https.

Regards,
-Justin


Yup, this specific issue is HTTPS specific (urllib3 does a case-sensitive comparison to `https`, which obviously fails).


@Lukasa Gotcha. Luckily, this is relatively easy to workaround in my case. Though, I wonder what would happen if Requests were configured to auto-follow redirects (while using proxies) and it encountered an HTTPS URL in the Location header (which is what would happen in my case, except I'm manually following each redirect, so I can check for the case and adjust it).

Anyway, thanks again for your help. :-)


@Lukasa Is there a way to workaround the issue in a custom HTTPAdapter subclass? I tried fixing the scheme in the add_headers method, but it looks like that doesn't catch the issue early enough in the flow (I haven't dug through the code enough yet to know exactly what's happening when).


@Lukasa Nvm, I overrode the get_connection method in my own HTTPAdapter child class to make any https schemes lower case before calling the parent class's get_connection method. :-)


@correcthorsebatterystaple- Yup, that'd be the best place to work around this for now. =)