https requests over proxy

Have run into an issue while trying to access https over proxy. Have had no issues doing the same via curl.

Here is a log of attempting to access https://github.com:

``` python
PROXY_LIST = {
    'http':'http://myproxy:80' ,
    'https':'http://myproxy:80'
}
connection = requests.get('https://github.com',proxies=PROXY_LIST)
print connection.content
```

Outputs: 

``` html
<html>
<head><title>400 The plain HTTP request was sent to HTTPS port</title></head>
<body bgcolor="white">
<center><h1>400 Bad Request</h1></center>
<center>The plain HTTP request was sent to HTTPS port</center>
<hr><center>nginx/1.0.4</center>
</body>
</html>
```

Any suggestions would be appreciated.


I too am getting hit by this issue. It appears that `requests` fails to send a CONNECT request for the SSL handshake before sending the GET request.


http://stackoverflow.com/questions/8287628/proxies-with-python-requests-module


http://stackoverflow.com/questions/8482896/making-http-requests-via-python-requests-module-not-working-via-proxy-where-curl


Quick note, the following works using urllib2:

```
import urllib2

url = 'https://google.com'
proxy = urllib2.ProxyHandler({'https': '1.2.3.4:5678'})
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)
request = urllib2.Request(url)
response = urllib2.urlopen(request)
```


To be clear, this is a bug.


@shazow, we need to send `CONNECT` first.


@kennethreitz For which? How do you implement proxies? Are you using ProxyManager?


@shazow https://github.com/kennethreitz/requests/blob/develop/requests/models.py#L447


@kennethreitz Thanks! Hmm. The underlying HTTPSConnectionPool should be making a CONNECT.

Are you sure you're actually setting it to HTTPS? Because if you look at the example...

``` python
PROXY_LIST = {
    'http':'http://myproxy:80' ,
    'https':'http://myproxy:80'
}
```

Both of the proxies are http and port 80, so urllib3 has no way of knowing it's HTTPS unless you do it explicitly.

Anyways, if it is urllib3's fault, if someone writes a failing test in urllib3, I'll look into it this weekend. :)

Something between...
https://github.com/shazow/urllib3/blob/master/test/with_dummyserver/test_https.py and
https://github.com/shazow/urllib3/blob/master/test/with_dummyserver/test_socketlevel.py


Ah, we're getting it from the scheme of the proxy, not the key in the dict. hmm.


From what I can gather, the problem apperas when trying to use HTTP proxy to connect to the HTTPS destination (at least that's my use case, I think it fits the description of this issue).

Apparently, httplib does have support for HTTP Tunneling (ie. CONNECT), which is needed to do HTTPS over HTTP, but it was added only recently, in 2.7 (which kind of decreases the usefullness the hacks described). But it's not used automatically, it needs to be specifically set using set_tunnel()

I tried to see how difficult it would be to hack the support in (modifying both urllib3 and requests), and ended up with the following:
- Modified requests.models to pass the destination URL to poolmanager.proxy_from_url()
- Modified poolmanager.proxy_from_url() to check whether the destination scheme is HTTPS, and if so, set a flag while calling connectionpool.connection_from_url()
- Modified connectionpool.connection_from_url() to check this flag, and if it's set, and the scheme for the connection itself is HTTP, use HTTPSConnectionPool nevertheless and set a flag to tell it it's HTTPS over HTTP
- Modified HTTPSConnectionPool._new_conn to check for this flag, and if it's HTTPS over HTTP, force usage of plain ol' HTTPSConnection instead of VerifiedHTTPSConnection
- Modified HTTPConnectionPool._make_request() to check that url is not the same host, and the url scheme is https (meaning we're tunneling HTTPS), and use httlib's set_tunnel() 

I have no idea whether the above steps are a horrible, horrible hack, or a good way to solve this, but it does work for me.

If it sounds sane, I can make patches for requests and urllib3 to properly implement it. If there's a better way, please tell me :)


@senko Hmm interesting. And this only works in 2.7 and silently passes in earlier versions?

I'd love to see the actual changes, would be great if you sent a pull request to urllib3. HTTPS over HTTP is not something I considered before, so it's something I'd like to look into with your help. :-)

Bonus points if you include an HTTPS over HTTP unit test... (Might be challenging)


(EDITED)

Hi guys,

We came up against this problem and have implemented a very bleeding edge fix, based on the original code from urllib2 - however it seems to be having issues.

Soon as we've got something stable, we'll re-post here.


Hi all,

Just to let you know, we've now added SSL and SOCKS proxy support into our latest fork.

If you want to grab it whilst it's hot, see the following:
https://github.com/kennethreitz/requests/pull/478

Hope this helps!

Cal


I believe this is fixed and can be closed based upon the merge of https://github.com/kennethreitz/requests/pull/478 

Simple smoke test

``` sh
pip install mitmproxy
mitmproxy
```

``` python
PROXY_LIST = { 'https':'http://localhost:8080' }
c = requests.get('https://github.com/', proxies=PROXY_LIST)
```

Observer connection in mitmproxy